Skip to main content

Advertisement

SpringerLink
Log in

S3D-CNN: skeleton-based 3D consecutive-low-pooling neural network for fall detection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Most existing deep-learning-based fall detection methods use either 2D neural network without considering movement representation sequences, or whole sequences instead of only those in the fall period. These characteristics result in inaccurate extraction of human action features and failure to detect falls due to background interferences or activity representation beyond the fall period. To alleviate these problems, a skeleton-based 3D consecutive-low-pooling neural network (S3D-CNN) for fall detection is proposed in this paper. In the S3D-CNN, an activity feature clustering selector is designed to extract the skeleton representation in depth videos using pose estimation algorithm and form optimized skeleton sequence of fall period. A 3D consecutive-low-pooling (3D-CLP) neural network is proposed to process these representation sequences by improving network in terms of layer number, pooling kernel size, and single input frame number. The proposed method is evaluated on public and self-collected datasets respectively, outperforming the existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Sannino G, De Falco I, De Pietro G (2015) A supervised approach to automatically extract a set of rules to support fall detection in an mHealth system. Appl Soft Comput 34:205–216. https://doi.org/10.1016/j.asoc.2015.04.060

    Article  Google Scholar 

  2. Lapierre N, Neubauer N, Miguel-Cruz A, Rincon AR, Liu L, Rousseau J (2018) The state of knowledge on technologies and their use for fall detection: a scoping review. Int J Med Inform 116:9–9. https://doi.org/10.1016/j.ijmedinf.2018.05.001

    Article  Google Scholar 

  3. Jokanovic B, Amin M (2018) Fall detection using deep learning in range-doppler radars. IEEE T Aero Elec Sys 54(1):180–189. https://doi.org/10.1109/Taes.2017.2740098

    Article  Google Scholar 

  4. Lu N, Wu YD, Feng L, Song JB (2019) Deep learning for fall detection: three-dimensional CNN combined with LSTM on video kinematic data. IEEE J Biomed Health 23(1):314–323. https://doi.org/10.1109/Jbhi.2018.2808281

    Article  Google Scholar 

  5. Cao Z, Simon T, Wei SE, Sheikh Y (2017) Realtime multi-person 2D pose estimation using part affinity fields. In: proceedings of the CVPR IEEE: 1302–1310. doi:https://doi.org/10.1109/Cvpr.2017.143

  6. Chen YL, Wang ZC, Peng YX, Zhang ZQ, Yu G, Sun J (2018) Cascaded pyramid network for multi-person pose estimation. In: proceedings of the 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR):7103–7112. doi:https://doi.org/10.1109/Cvpr.2018.00742

  7. Fang HS, Xie SQ, Tai YW, Lu CW (2017) RMPE: Regional Multi-Person Pose Estimation. In: Proceedings of the IEEE Conf Comp Vis: 2353–2362. doi:https://doi.org/10.1109/ICCV.2017.256

  8. Guler RA, Neverova N, Kokkinos L (2018) DensePose: dense human pose estimation in the wild. In: proceedings of the 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR): 7297–7306. doi:https://doi.org/10.1109/Cvpr.2018.00762

  9. Liang XD, Gong K, Shen XH, Lin L (2019) Look into person: Joint Body Parsing & Pose Estimation Network and a new benchmark. IEEE T Pattern Anal 41(4):871–885. https://doi.org/10.1109/Tpami.2018.2820063

    Article  Google Scholar 

  10. Wei W, Qi Y (2011) Information potential fields navigation in wireless ad-hoc sensor networks. Sensors 11(5):4794–4807. https://doi.org/10.3390/s110504794

    Article  Google Scholar 

  11. Wei W, Xu Q, Wang L, Hei XH, Shen P, Shi W, Shan L (2013) GI/Geom/1 queue based on communication model for mesh networks. Int J Commun Syst 27(11):3013–3029. https://doi.org/10.1002/dac.2522

    Article  Google Scholar 

  12. Wei W, Yang X, Shen P, Zhou B (2012) Holes detection in anisotropic sensornets: topological methods. International Journal of Distributed Sensor Networks 2012:1–9. https://doi.org/10.1155/2012/135054

    Article  Google Scholar 

  13. Wei W, Song H, Li W, Shen P, Vasilakos A (2017) Gradient-driven parking navigation using a continuous information potential field based on wireless sensor network. Information Science 408:100–114. https://doi.org/10.1016/j.ins.2017.04.042

    Article  Google Scholar 

  14. Khojasteh SB, Villar JR, Chira C, Gonzalez VM, de la Cal E (2018) Improving fall detection using an on-wrist wearable accelerometer. Sensors-Basel 18 (5). doi:https://doi.org/10.3390/S18051350

  15. Pang I, Okubo Y, Sturnieks D, Lord SR, Brodie MA (2019) Detection of near falls using wearable devices: a systematic review. J Geriatr Phys Ther 42(1):48–56. https://doi.org/10.1519/Jpt.0000000000000181

    Article  Google Scholar 

  16. Veeraputhiran A, Sankararajan R (2019) Feature based fall detection system for elders using compressed sensing in WVSN. Wirel Netw 25(1):287–301. https://doi.org/10.1007/s11276-017-1557-3

    Article  Google Scholar 

  17. Ahn S, Kim J, Koo B, Kim Y (2019) Evaluation of inertial sensor-based pre-impact fall detection algorithms using public dataset. Sensors-Basel 19 (4). doi:https://doi.org/10.3390/S19040774

  18. Ma C, Shimada A, Uchiyama H, Nagahara H, Taniguchi R (2019) Fall detection using optical level anonymous image sensing system. Opt Laser Technol 110:44–61. https://doi.org/10.1016/j.optlastec.2018.07.013

    Article  Google Scholar 

  19. Yu S, Chen H, Brown RA (2018) Hidden Markov mode based fall detection with motion sensor orientation calibration: a case for real-life home monitoring. IEEE J Biomed Health 22(6):1847–1853. https://doi.org/10.1109/Jbhi.2017.2782079

    Article  Google Scholar 

  20. Nizam Y, Mohd MNH, Jamil MMA (2018) Development of a user-adaptable human fall detection based on fall risk levels using depth sensor. Sensors-Basel 18 (7). doi:https://doi.org/10.3390/S18072260

  21. Luo XM, Liu T, Liu J, Guo XM, Wang GL (2012) Design and implementation of a distributed fall detection system based on wireless sensor networks. Eurasip J Wirel Comm 1–13. doi:https://doi.org/10.1186/1687-1499-2012-118

  22. Bourke AK, O'Brien JV, Lyons GM (2007) Evaluation of a threshold-based tri-axial accelerometer fall detection algorithm. Gait Posture 26(2):194–199. https://doi.org/10.1016/j.gaitpost.2006.09.012

    Article  Google Scholar 

  23. Makhlouf A, Nedjai I, Saadia N, Ramdane-Cherif A (2017) Multimodal system for fall detection and location of person in an intelligent habitat. Procedia Comput Sci 109:969–974. https://doi.org/10.1016/j.procs.2017.05.455

    Article  Google Scholar 

  24. Yang L, Ren YY, Hu HS, Tian B (2015) New fast fall detection method based on spatio-temporal context tracking of head by using depth images. Sensors-Basel 15(9):23004–23019. https://doi.org/10.3390/s150923004

    Article  Google Scholar 

  25. Panahi L, Ghods V (2018) Human fall detection using machine vision techniques on RGB-D images. Biomed Signal Proces 44:146–153. https://doi.org/10.1016/j.bspc.2018.04.014

    Article  Google Scholar 

  26. Abobakr A, Hossny M, Nahavandi S (2018) A skeleton-free fall detection system from depth images using random decision Forest. IEEE Syst J 12(3):2994–3005. https://doi.org/10.1109/Jsyst.2017.2780260

    Article  Google Scholar 

  27. Li X, Nie LS, Xu HC, Wang XZ (2018) Collaborative fall detection using smart phone and Kinect. Mobile Netw Appl 23(4):775–788. https://doi.org/10.1007/s11036-018-0998-y

    Article  Google Scholar 

  28. Thome N, Miguet S, Ambellouis S (2008) A real-time, multiview fall detection system: a LHMM-based approach. IEEE T Circ Syst Vid 18(11):1522–1532. https://doi.org/10.1109/Tcsvt.2008.2005606

    Article  Google Scholar 

  29. Zhang CL, Wu JX (2019) Improving CNN linear layers with power mean non-linearity. Pattern Recogn 89:12–21. https://doi.org/10.1016/j.patcog.2018.12.029

    Article  Google Scholar 

  30. Ren SQ, He KM, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE T Pattern Anal 39(6):1137–1149. https://doi.org/10.1109/Tpami.2016.2577031

    Article  Google Scholar 

  31. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR):779–788. doi:https://doi.org/10.1109/Cvpr.2016.91

  32. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) SSD: single shot MultiBox detector. Lect Notes Comput Sc 9905:21–37. https://doi.org/10.1007/978-3-319-46448-0_2

    Article  Google Scholar 

  33. Min WD, Cui H, Rao H, Li ZX, Yao LY (2018) Detection of human falls on furniture using scene analysis based on deep learning and activity characteristics. IEEE Access 6:9324–9335. https://doi.org/10.1109/Access.2018.2795239

    Article  Google Scholar 

  34. Zhang ZM, Ma X, Wu HB, Li YB (2019) Fall detection in videos with trajectory-weighted deep-convolutional rank-pooling descriptor. IEEE Access 7:4135–4144. https://doi.org/10.1109/Access.2018.2887144

    Article  Google Scholar 

  35. Petersen NC, Rodrigues F, Pereira FC (2019) Multi-output bus travel time prediction with convolutional LSTM neural network. Expert Syst Appl 120:426–435. https://doi.org/10.1016/j.eswa.2018.11.028

    Article  Google Scholar 

  36. Greff K, Srivastava RK, Koutnik J, Steunebrink BR, Schmidhuber J (2017) LSTM: a search space odyssey. IEEE T Neur Net Lear 28(10):2222–2232. https://doi.org/10.1109/Tnnls.2016.2582924

    Article  MathSciNet  Google Scholar 

  37. Liu HF, Liu D, Sun XY, Wu F, Zeng WJ (2017) On-line fall detection via a boosted Cascade of hybrid features. In: Proceedings of the IEEE Int Conf Multi 100–105. doi:https://doi.org/10.1109/ICMEW.2017.8026316

  38. Lotfi A, Albawendi S, Powell H, Appiah K, Langensiepen C (2018) Supporting independent living for older adults; employing a visual based fall detection through Analysing the motion and shape of the human body. IEEE Access 6:70272–70282. https://doi.org/10.1109/Access.2018.2881237

    Article  Google Scholar 

  39. Min WD, Zou S, Li J (2019) Human fall detection using normalized shape aspect ratio. Multimed Tools Appl 78(11):14331–14353. https://doi.org/10.1007/s11042-018-6794-7

    Article  Google Scholar 

  40. Abeyruwan SW, Sarkar D, Sikder F, Visser U (2016) Semi-automatic extraction of training examples from sensor readings for fall detection and posture monitoring. IEEE Sensors J 16(13):5406–5415. https://doi.org/10.1109/Jsen.2016.2559804

    Article  Google Scholar 

  41. Nizam Y, Mohd MNH, Jamil MMA (2017) Human fall detection from depth images using position and velocity of subject. In: proceedings of the 2016 IEEE international symposium on robotics and intelligent sensors 105:131–137. doi:https://doi.org/10.1016/j.procs.2017.01.191

  42. Mastorakis G, Ellis T, Makris D (2018) Fall detection without people: a simulation approach tackling video data scarcity. Expert Syst Appl 112:125–137. https://doi.org/10.1016/j.eswa.2018.06.019

    Article  Google Scholar 

  43. Fan Y, Levine MD, Wen G, Qiu S (2017) A deep neural network for real-time detection of falling humans in naturally occurring scenes. Neurocomputing. 260:43–58. https://doi.org/10.1016/j.neucom.2017.02.082

    Article  Google Scholar 

  44. Min WD, Yao LY, Lin ZR, Liu L (2018) Support vector machine approach to fall recognition based on simplified expression of human skeleton action and fast detection of start key frame using torso angle. IET Comput Vis 12(8):1133–1140. https://doi.org/10.1049/iet-cvi.2018.5324

    Article  Google Scholar 

  45. Min WD, Cui H, Han Q, Zou FY (2018) A scene recognition and semantic analysis approach to unhealthy sitting posture detection during screen-Reading. Sensors-Basel 18 (9). doi: https://doi.org/10.3390/S18093119

  46. Melzer I, Benjuya N, Kaplanski J, Alexander N (2009) Association between ankle muscle strength and limit of stability in older adults. Age Ageing 38(1):119–123. https://doi.org/10.1093/ageing/afn249

    Article  Google Scholar 

  47. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3D convolutional networks. In: proceedings of the 2015 IEEE international conference on computer vision (ICCV):4489–4497. doi:https://doi.org/10.1109/Iccv.2015.510

  48. Sung JY, Ponce C, Selman B, Saxena A (2012) Unstructured human activity detection from RGBD images. In: Proceedings of the IEEE Int Conf Robot:842–849. doi:https://doi.org/10.1109/ICRA.2012.6224591

  49. Kwolek B, Kepski M (2014) Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput Meth Prog Bio 117(3):489–501. https://doi.org/10.1016/j.cmpb.2014.09.005

    Article  Google Scholar 

  50. Charfi I, Miteran J, Dubois J, Atri M, Tourki R (2012) Definition And Performance Evaluation Of A Robust SVM Based Fall Detection Solution. In: Proceedings of the 8th International Conference on Signal Image Technology & Internet Based Systems:218–224. doi:https://doi.org/10.1109/SITIS.2012.155

  51. Auvinet E, Multon F, Saint-Arnaud A, Rousseau J, Meunier J (2011) Fall detection with multiple cameras: an occlusion-resistant method based on 3-D Silhouette vertical distribution. IEEE T Inf Technol B 15(2):290–300. https://doi.org/10.1109/Titb.2010.2087385

    Article  Google Scholar 

  52. Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T (2011) HMDB: a large video database for human motion recognition. In: proceedings of the 2011 IEEE international conference on computer vision (ICCV):2556–2563

  53. Soomro K, Zamir AR, Subarak M (2012) UCF101: A dataset of 101 human actions classes from videos in the wild, arXiv: 1212.0402

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 61762061), the Natural Science Foundation of Jiangxi Province, China (Grant No. 20161ACB20004) and Jiangxi Key Laboratory of Smart City (Grant No. 20192BCD40002).

Author information

Authors and Affiliations

  1. School of Information Engineering, Nanchang University, Nanchang, 330031, China

    Xin Xiong, Pin Liao, Hao Yang & Shuai Wang

  2. School of Software, Nanchang University, Nanchang, 330047, China

    Weidong Min

  3. Jiangxi Key Laboratory of Smart City, Nanchang, 330047, China

    Weidong Min

  4. School of Data and Computer Science, Sun Yat-sen University, Guangzhou, 510006, China

    Wei-Shi Zheng

Authors
  1. Xin Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weidong Min
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei-Shi Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pin Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shuai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weidong Min.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiong, X., Min, W., Zheng, WS. et al. S3D-CNN: skeleton-based 3D consecutive-low-pooling neural network for fall detection. Appl Intell 50, 3521–3534 (2020). https://doi.org/10.1007/s10489-020-01751-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01751-y

Keywords

Search

Navigation