Skip to main content
SpringerLink
Log in

Fall detection based on fused saliency maps

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Fall detection is drawing more attention from both academia and industry. The human body occupies smaller space relative to the background in images, so the complex background affects the extraction of human fall or non-fall features. In order to reduce the interference of the complex background, a fall detection method based on fused saliency maps is proposed, which consists of saliency maps generation model and fall detection model. For saliency maps generation model, M-level segmentation is to obtain segmented images in different level. The saliency detection mainly uses two-stream convolutional neural network extract global and local features to generate the saliency maps. The saliency maps fusion automatically learns the weights according to mean structural similarity for fusing saliency maps. For fall detection model, a simple deep network is constructed to extract the discriminant features of fall or non-fall, where the fused saliency maps is used. Experimental result show that the proposed method achieves 99.67% and 98.92% accuracy on UR Fall Detection and our self-built NT Fall Detection database, respectively. And the convergence speed is fastest compared with those of using RGB images and depth images. The proposed fall detection method that can reduce the interference of complex background outperforms the other methods in terms of higher accuracy and faster convergence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Abualigah LMQ (2019) Feature selection and enhanced krill herd algorithm for text document clustering. Springer, Berlin

    Book  Google Scholar 

  2. Abualigah LMQ, Khader AT, Hanandeh ES (2018) Hybrid clustering analysis using improved krill herd algorithm. Appl Intell 48:4047–4071

    Article  Google Scholar 

  3. Alexis BP, Scott RK (2020) Deaths from fall-related traumatic brain injury-United States, 2008-2017. Morb Mortal Wkly Rep 69(9):225–230

    Google Scholar 

  4. Alzahrani MS, Jarraya SK, Ben-Abdallah H, Ali MS (2019) Comprehensive evaluation of skeleton features-based fall detection from Microsoft Kinect v2. Signal Image Vid Process 13(7):1431–1439

    Article  Google Scholar 

  5. Bet P, Castro PC, Ponti MA (2019) Fall detection and fall risk assessment in older person using wearable sensors: a systematic review. Int J Med Inform 130:103946

    Article  Google Scholar 

  6. Cai X, Li S, Liu X, Han G (2020) Vision-based fall detection with multi-task hourglass convolutional auto-encoder. IEEE Access 8:44493–44502

    Article  Google Scholar 

  7. Casilari-Pérez E, García-Lagos F (2019) A comprehensive study on the use of artificial neural networks in wearable fall detection systems. Expert Syst Appl 138:112811

    Article  Google Scholar 

  8. Chen Y, Wang J, Liu S, Chen X, Xiong J, Xie J, Yang K (2019) Multiscale fast correlation filtering tracking algorithm based on a feature fusion model. Concurr Comput-Pract Exp. https://onlinelibrary. wiley.com/doi/abs/10.1002/cpe.5533

  9. Chen Y, Xu W, Zuo J, Yang K (2019) The fire recognition algorithm using dynamic feature fusion and IV-SVM classifier. Clust Comput 22:S7665–S7675

    Article  Google Scholar 

  10. Chen Y, Tao J, Zhang Q, Yang K, Chen X, Xiong J, Xia R, Xie J (2020) Saliency detection via the improved hierarchical principal component analysis method. Wirel Commun Mob Comput 2020:1–12. https://doi.org/10.1155/2020/8822777

    Article  Google Scholar 

  11. Chen Y, Tao J, Liu L, Xiong J, Xia R, Xia J, Zhang Q, Yang K (2020) Research of improving semantic image segmentation based on a feature fusion model. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-020-02066-z

  12. Chung KL, Liu LT, Liao CH (2019) Novel and robust vision- and system-on-chip- based sensor for fall detection. Sensors Mater 31(8):2657–2668

    Article  Google Scholar 

  13. Dhiman C, Vishwakarma DK (2020) View-invariant deep architecture for human action recognition using two-stream motion and shape temporal dynamics. IEEE Tran Image Process 29:3835–3844

    Article  Google Scholar 

  14. Ezatzadeh S, Keyvanpour MR (2019) ViFa: an analytical framework for vision-based fall detection in a surveillance environment. Multimed Tools Appl 78(18):25515–25537

    Article  Google Scholar 

  15. Ge C, Gu YH, Yang J (2017) Human fall detection using segment-level CNN features and sparse dictionary learning. In: IEEE International Workshop on Machine Learning for Signal Processing, 1–6

  16. Ge CJ, Gu IYH, Yang J (2018) Co-saliency-enhanced deep recurrent convolutional networks for human fall detection in E-healthcare. In: IEEE Engineering in Medicine and Biology Society, 1572–1575

  17. Harrou F, Zerrouki N, Sun Y, Houacine A (2017) Vision-based fall detection system for improving safety of elderly people. IEEE Instrum Meas Mag 20(6):49–55

    Article  Google Scholar 

  18. Hasan MM, Islam MS, Abdullah S (2019) Robust pose-based human fall detection using recurrent neural network. In: International. Conference on Robotics, Automation, Artificial-intelligence and Internet-of-Things, 48–51

  19. Huo SW, Zhou Y, Xiang W (2019) Semisupervised learning based on a novel iterative optimization model for saliency detection. IEEE Trans Neural Netw Learn Syst 30(1):225–241

    Article  Google Scholar 

  20. Jansi R, Amutha R (2020) Detection of fall for the elderly in an indoor environment using a tri-axial accelerometer and Kinect depth data. Multidim Syst Sign Process. https://doi.org/10.1007/s11045-020-00705-4

  21. Khraief C, Benzarti F, Amiri H (2020) Elderly fall detection based on multi-stream deep convolutional networks. Multimed Tools Appl 79:19537–19560. https://doi.org/10.1007/s11042-020-08812-x

    Article  Google Scholar 

  22. Kong Y, Huang J, Huang S, Wei Z, Wang S (2019) Learning spatiotemporal representations for human fall detection in surveillance video. J Vis Commun Image Represent 59:215–230

    Article  Google Scholar 

  23. Kwolek B, Kepski M (2014) Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput Methods Prog Biomed 117:489–501

    Article  Google Scholar 

  24. Kwolek B, Kepski M (2015) Improving fall detection by the use of depth sensor and accelerometer. Neurocomputing 168:637–645

    Article  Google Scholar 

  25. Leite G, Silva G, Pedrini H (2019) Fall detection in video sequences based on a three-stream convolutional neural network. In: IEEE International Conference On Machine Learning And Application, 191–195

  26. Li GY, Yu Y (2015) Visual saliency based on multiscale deep features. In: IEEE Conference on Computer Vision and Pattern Recognition, 5455–5463

  27. Li C, Li H, Ding Y, Xu K, Zhang W (2017) Joint object location and deep feature learning for fall recognition. In: International Congress on Image and Signal Processing, BioMedical Engineering and Informatics https://doi.org/10.1109/CISP-BMEI.2017.8302065

  28. Li X, Yang F, Cheng H, Liu W (2018) Contour knowledge transfer for salient object detection. In: European Conference on Computer Vision, 355–370

  29. Li JX, Rajan D, Yang J (2018) Locality and context-aware top-down saliency. IET Image Process 12(3):400–407

    Article  Google Scholar 

  30. Lie WN, Hsu FY, Hsu YL (2019) Fall-down event detection for elderly based on motion history images and deep learning. Int Workshop Advanced Image Technol 11049: UNSP 110493Z

  31. Liu W, Guo J, Huang Z, Qiu D (2017) Falling-action analysis algorithm based on convolutional neural network. In: International Conference on Communication and Electronic Information Engineering, 37–42

  32. Liu J, Hou Q, Cheng MM, Feng J, Jiang J (2019) A simple pooling-based design for real-time salient object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, 3912–3921

  33. Lu N, Wu Y, Feng L, Song J (2019) Deep learning for fall detection: three-dimensional CNN combined with LSTM on video kinematic data. IEEE J Biomed Health Inform 23(1):314–323

    Article  Google Scholar 

  34. Luvizon DC, Tabia H, Picard D (2019) Multi-task deep learning for real-time 3D human pose estimation and action recognition. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2020.2976014

  35. Min WD, Yao LY, Lin ZR (2018) Support vector machine approach to fall recognition based on simplified expression of human skeleton action and fast detection of start key frame using torso angle. IET Comput Vis 12(8):1133–1140

    Article  Google Scholar 

  36. Núñez-Marcos A, Azkune G, Arganda-Carreras I (2017) Vision-based fall detection with convolutional neural networks. Wirel Commun Mob Comput 1:1–16

    Article  Google Scholar 

  37. Qiu Y, Liu Y, Yang H, Xu J (2020) A simple saliency detection approach via automatic top-down feature fusion. Neurocomputing 388:124–134

    Article  Google Scholar 

  38. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Neural Information Processing Systems, 568–576

  39. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The Asilomar Conference on Signals, Systems and Computers, 1398–1402

  40. Wang J, Jiang H, Yuan Z, Cheng MM, Hu X, Zheng N (2017) Salient object detection: a discriminative regional feature integrationapproach. Int J Comput Vis 123:251–268

    Article  Google Scholar 

  41. Wang W, Shen J, Cheng MM, Shao N (2019) An iterative and cooperative top-down and bottom-up inference network for salient object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, 5961–5970

  42. Yang R, Wang W, Lai Q, Fu H (2019) Salient object detection in the deep learning era: an in-depth survey. arXiv:1904.09146.

  43. Yun Y, Gu YH (2015) Human fall detection via shape analysis on Riemannian manifolds with applications to elderly care. In: IEEE International Conference on Image Processing, 3280–3284

  44. Zerrouki N, Houacine A (2018) Combined curvelets and hidden Markov models for human fall detection. Multimed Tools Appl 77(5):6405–6424

    Article  Google Scholar 

  45. Zhang D, Fu H, Han J, Borji A, Li X (2018) A review of co-saliency detection algorithms: fundamentals, applications, and challenges. ACM Trans Intell Syst Technol 9:1–31. https://doi.org/10.1145/3158674

    Article  Google Scholar 

  46. Zhang L, Zhang D, Sun J, Wei G, Bo H (2019) Salient object detection by local and global manifold regularized SVM model. Neurocomputing 340:42–54

    Article  Google Scholar 

  47. Zhang LG, Sheng Z, Li YB (2020) Image object detection and semantic segmentation based on convolutional neural network. Neural Computing Appl 32(7):1949–1958

    Article  Google Scholar 

  48. Zhao T, Wu X (2019) Pyramid feature selective network for saliency detection In: IEEE Conference on Computer Vision and Pattern Recognition arXiv: 1903.00179

Download references

Acknowledgments

This work is supported by National Natural Science Foundation of China (NO.61871241, NO.61976120);the Science and Technology Program of Nantong (JC2018025, JC2018129); Nantong University-Nantong Joint Research Center for Intelligent Information Technology (KFKT2017B04); Nanjing University State Key Lab. for Novel Software Technology (KFKT2019B15); Postgraduate Research and Practice Innovation Program of Jiangsu Province (KYCX19_2056).

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Nantong University, 9 Seyuan road, Nantong, 226019, Jiangsu, People’s Republic of China

    Hongjun Li & Chaobo Li

  2. State Key Lab. for Novel Software Technology, Nanjing University, Nanjing, 210023, Jiangsu, People’s Republic of China

    Hongjun Li

  3. Nantong Research Institute for Advanced Communication Technologies, Nantong, 226019, Jiangsu, People’s Republic of China

    Hongjun Li

  4. TONGKE School of Microelectronics, Nantong, 226019, Jiangsu, People’s Republic of China

    Hongjun Li

  5. China Mobile Communications Group Co., Ltd., Taizhou, Jiangsu, 225300, People’s Republic of China

    Yupeng Ding

Authors
  1. Hongjun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chaobo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yupeng Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongjun Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Li, C. & Ding, Y. Fall detection based on fused saliency maps. Multimed Tools Appl 80, 1883–1900 (2021). https://doi.org/10.1007/s11042-020-09708-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09708-6

Keywords

Search

Navigation