Skip to main content

Advertisement

SpringerLink
Log in

Deep-learning-based human activity recognition for Alzheimer’s patients’ daily life activities assistance

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Alzheimer’s disease is considered as one of the most well-known illnesses in the elderly. It is a neurodegenerative and irreversible brain disorder that slowly destroys memory, thinking ability, and ultimately the ability to perform even basic daily tasks. In fact, people suffering from this disorder have difficulty remembering events, recognizing objects and faces, remembering the meaning of words, and developing judgment. As a result, their cognitive abilities are impaired and they are unable to perform activities of daily living independently. Therefore, patients need constant support to carry out their daily activities. In this study, we propose a new support system to support patients with Alzheimer’s disease to carry out their daily tasks independently. The proposed assistance systems are composed of two parts. The first is a human activity recognition (HAR) module to monitor the patient behaviour. Here, we proposed two HAR systems. The first is based on 2D skeleton data and convolution neural network, and the second is based on 3D skeleton and transformers. The second part of the assistance systems consists of a support module that recognizes the patient’s behavioural abnormalities and issues appropriate warnings. Here, we also proposed two methods. The first is based on a simple conditional structure, and the second is based on a reinforcement learning technique. As a result, we obtain four different assistance systems for Alzheimer’s patients. Finally, a comparative study between the four systems was carried out in terms of performance and time complexity using the DemCare dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. https://colab.research.google.com/.

  2. https://www.python.org/.

  3. https://keras.io/.

  4. https://demcare.eu/datasets/.

  5. https://github.com/ibaiGorordo/ONNX-Mobile-Human-Pose-3D.

References

  1. Division UP (2019) World population ageing, 2019: highlights, p 37

  2. Fuster V (2017) Changing demographics. J Am Coll Cardiol 69(24):3002–3005. https://doi.org/10.1016/j.jacc.2017.05.013

    Article  Google Scholar 

  3. Chernbumroong S, Cang S, Atkins A, Yu H (2013) Elderly activities recognition and classification for applications in assisted living. Expert Syst Appl 40(5):1662–1674. https://doi.org/10.1016/j.eswa.2012.09.004

    Article  Google Scholar 

  4. Wild S, Roglic G, Green A, Sicree R, King H (2004) Global prevalence of diabetes. Diabetes Care 27(5):1047–1053. https://doi.org/10.2337/diacare.27.5.1047

    Article  Google Scholar 

  5. Association A (2016) 2016 Alzheimer’s disease facts and figures. Alzheimer’s Dement 12(4):459–509. https://doi.org/10.1016/j.jalz.2016.03.001

    Article  Google Scholar 

  6. Jackson-Webb FBF, Duckett RS Moodie: Australia’s health 2016 report card: Australians living longer but with more chronic disease. https://www.aihw.gov.au/getmedia/9844cefb-7745-4dd8-9ee2-f4d1c3d6a727/19787-AH16.pdf.aspx

  7. Dua T, Seeher K, Sivananthan S, Chowdhary N, Pot A, Saxena S (2017) World Health Organization’s global action plan on the public health response to dementia 2017–2025. Alzheimer’s Dement 13:1450–1451. https://doi.org/10.1016/j.jalz.2017.07.758

    Article  Google Scholar 

  8. Kwak S, Han B, Han JH (2011) Scenario-based video event recognition by constraint flow, pp 3345–3352. https://doi.org/10.1109/CVPR.2011.5995435

  9. Gaur U, Zhu Y, Song B, Roy-Chowdhury A (2011) A string of feature graphs model for recognition of complex activities in natural videos. In: 2011 international conference on computer vision, pp 2595–2602. https://doi.org/10.1109/ICCV.2011.6126548

  10. Duric Z, Gray WD, Heishman R, Fayin Li, Rosenfeld A, Schoelles MJ, Schunn C, Wechsler H (2002) Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction. Proc IEEE 90(7):1272–1289. https://doi.org/10.1109/JPROC.2002.801449

    Article  Google Scholar 

  11. Thangali A, Nash JP, Sclaroff S, Neidle C (2011) Exploiting phonological constraints for handshape inference in ASL video. In: CVPR 2011, pp 521–528. https://doi.org/10.1109/CVPR.2011.5995718

  12. Hankyu M, Rajeev S, Namsoon J (2012) Method and system for measuring shopper response to products based on behavior and facial expression. https://lens.org/105-447-594-886-96X

  13. Johansson G (1975) Visual motion perception. Sci Am 232(6):76–88

    Article  Google Scholar 

  14. Campbell LW, Bobick AF (1995) Recognition of human body motion using phase space constraints. In: Proceedings of IEEE international conference on computer vision, pp 624–630. https://doi.org/10.1109/ICCV.1995.466880

  15. Sheikh Y, Sheikh M, Shah M (2005) Exploring the space of a human action. In: Tenth IEEE international conference on computer vision (ICCV’05), vol 1, pp 144–149. https://doi.org/10.1109/ICCV.2005.90

  16. Yilmaz A, Shah M (2005) Recognizing human actions in videos acquired by uncalibrated moving cameras. In: Tenth IEEE international conference on computer vision (ICCV’05), vol 1, pp 150–157. https://doi.org/10.1109/ICCV.2005.201

  17. Felzenszwalb PF, Huttenlocher DP (2005) Pictorial structures for object recognition. Int J Comput Vis 61(1):55–79. https://doi.org/10.1023/B:VISI.0000042934.15159.49

    Article  Google Scholar 

  18. Cao Z, Hidalgo G, Simon T, Wei S, Sheikh Y (2021) Openpose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans Pattern Anal Mach Intell 43(01):172–186. https://doi.org/10.1109/TPAMI.2019.2929257

    Article  Google Scholar 

  19. Papandreou G, Zhu T, Chen L-C, Gidaris S, Tompson J, Murphy K (2018) Personlab: person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds) Computer vision—ECCV 2018. Springer, Cham, pp 282–299

    Chapter  Google Scholar 

  20. Devanne M, Wannous H, Berretti S, Pala P, Daoudi M, Del Bimbo A (2013) Space-time pose representation for 3D human action recognition, pp 456–464. https://doi.org/10.1007/978-3-642-41190-8_49

  21. Gan L, Chen F-T (2013) Human action recognition using APJ3D and random forests. J Softw 8:2238–2245

    Article  Google Scholar 

  22. Wang J, Liu Z, Wu Y, Yuan J (2012) Mining actionlet ensemble for action recognition with depth cameras. In: 2012 IEEE conference on computer vision and pattern recognition, pp 1290–1297

  23. Xia L, Chen C-C, Aggarwal JK (2012) View invariant human action recognition using histograms of 3D joints. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops, pp 20–27

  24. Yang X, Tian Y (2014) Effective 3D action recognition using eigenjoints. J Vis Commun Image Represent 25(1):2–11. https://doi.org/10.1016/j.jvcir.2013.03.001 Visual Understanding and Applications with RGB-D Cameras

  25. Kim H, Kim I (2015) Human activity recognition as time-series analysis. Math Probl Eng 2015:1–9. https://doi.org/10.1155/2015/676090

    Article  Google Scholar 

  26. Taha A, Zayed HH, Khalifa ME, El-Horbaty E-SM (2015) Human activity recognition for surveillance applications. In: ICIT 2015, pp 577–586

  27. Gaglio S, Re GL, Morana M (2015) Human activity recognition process using 3-d posture data. IEEE Trans Hum Mach Syst 45:586–597

    Article  Google Scholar 

  28. Zhu Y, Lan Z, Newsam S, Hauptmann A (2019) Hidden two-stream convolutional networks for action recognition. In: Jawahar CV, Li H, Mori G, Schindler K (eds) Computer vision—ACCV 2018. Springer, Cham, pp 363–378

    Chapter  Google Scholar 

  29. Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231. https://doi.org/10.1109/TPAMI.2012.59

    Article  Google Scholar 

  30. Taylor GW, Fergus R, LeCun Y, Bregler C (2010) Convolutional Learning of spatio-temporal features. In: Proceedings of the 11th European conference on computer vision: Part VI. ECCV’10, pp 140–153. Springer, Berlin, Heidelberg. https://doi.org/10.5555/1888212.1888225

  31. Shamsipour G, Pirasteh S (2019) Artificial intelligence and convolutional neural network for recognition of human interaction by video from drone. https://doi.org/10.20944/preprints201908.0289.v1

  32. Bilen H, Fernando B, Gavves E, Vedaldi A (2018) Action recognition with dynamic image networks. IEEE Trans Pattern Anal Mach Intell 40(12):2799–2813. https://doi.org/10.1109/TPAMI.2017.2769085

    Article  Google Scholar 

  33. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. CoRR arXiv:1704.04861

  34. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  35. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCun Y (eds) ICLR. arXiv:1409.1556

  36. Du Y, Wang W, Wang L (2015) Hierarchical recurrent neural network for skeleton based action recognition. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1110–1118

  37. Zhu Y, Chen W, Guo G (2013) Fusing spatiotemporal features and joints for 3D action recognition. In: 2013 IEEE conference on computer vision and pattern recognition workshops, pp 486–491

  38. Liu J, Shahroudy A, Xu D, Kot AC, Wang G (2018) Skeleton-based action recognition using spatio-temporal LSTM network with trust gates. IEEE Trans Pattern Anal Mach Intell 40:3007–3021

    Article  Google Scholar 

  39. Noori FM, Wallace B, Uddin MZ, Torresen J (2019) A robust human activity recognition approach using openpose, motion features, and deep recurrent neural network. In: Felsberg M, Forssén P-E, Sintorn I-M, Unger J (eds) Image Anal. Springer, Cham, pp 299–310

    Chapter  Google Scholar 

  40. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Lu, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol 30, pp. 1–11

  41. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M., Heigold G, Gelly S, Uszkoreit J, Houlsby N (2020) An image is worth 16x16 words: transformers for image recognition at scale. CoRR arXiv:2010.11929

  42. Lin C-H, Yumer E, Wang O, Shechtman E, Lucey S (2018) St-gan: spatial transformer generative adversarial networks for image compositing. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 9455–9464

  43. Berg A, O’Connor M, Cruz MT (2021) Keyword transformer: a self-attention model for keyword spotting. In: Proc. Interspeech 2021, pp 4249–4253. https://doi.org/10.21437/Interspeech.2021-1286

  44. Jean-Baptiste E, Mihailidis A (2017) Benefits of automatic human action recognition in an assistive system for people with dementia. In: 2017 IEEE Canada international humanitarian technology conference (IHTC), pp 61–65

  45. Peters C, Hermann T, Wachsmuth S, Hoey J (2014) Automatic task assistance for people with cognitive disabilities in brushing teeth—a user study with the tebra system. ACM Trans Access Comput 5(4):1–34. https://doi.org/10.1145/2579700

    Article  Google Scholar 

  46. Jean-Baptiste EMD, Rotshtein P, Russell M (2016) Cogwatch: automatic prompting system for stroke survivors during activities of daily living. J Innov Digit Ecosyst 3(2):48–56. https://doi.org/10.1016/j.jides.2016.10.003

    Article  Google Scholar 

  47. Chen H, Soh Y (2018) A cooking assistance system for patients with Alzheimers disease using reinforcement learning. Int J Inf Technol 23(2), pp. 1–11

  48. Hou R, Chen C, Shah M (2017) An end-to-end 3D convolutional neural network for action detection and segmentation in videos. CoRR arXiv:1712.01111

  49. Hou R, Chen C, Sukthankar R, Shah M (2019) An efficient 3D CNN for action/object segmentation in video. CoRR arXiv:1907.08895

  50. Fernando B, Gavves E, José Oramas M, Ghodrati A, Tuytelaars T (2015) Modeling video evolution for action recognition. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 5378–5387. https://doi.org/10.1109/CVPR.2015.7299176

  51. Smola AJ, Schölkopf B (2003) A tutorial on support vector regression. Technical report, statistics and computing. https://doi.org/10.1023/B:STCO.0000035301.49549.88

  52. Hendrycks D, Gimpel K (2016) Bridging nonlinearities and stochastic regularizers with Gaussian error linear units. CoRR arXiv:1606.08415

  53. Karakostas A, Briassouli A, Avgerinakis K, Kompatsiaris I, Tsolaki M (2016) The dem@care experiments and datasets: a technical report

  54. Avgerinakis K, Briassouli A, Kompatsiaris Y (2016) Activity detection using sequential statistical boundary detection (SSBD). Comput Vis Image Underst 144:46–61

    Article  Google Scholar 

  55. Poularakis S, Avgerinakis K, Briassouli A, Kompatsiaris Y (2017) Efficient motion estimation methods for fast recognition of activities of daily living. Signal Process Image Commun 53:1–12

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the financial support of this work by grants from General Direction of Scientific Research (DGRST), Tunisia, under the ARUB program.

Funding

The authors have no relevant financial or non-financial interests to disclose.

Author information

Authors and Affiliations

  1. Research Team in Intelligent Machines (RTIM), National Engineering School of Gabes (ENIG), University of Gabes, 6029, Gabès, Tunisia

    Ahmed Snoun, Tahani Bouchrika & Olfa Jemai

Authors
  1. Ahmed Snoun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tahani Bouchrika
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olfa Jemai
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by AS. The first draft of the manuscript was written by AS, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ahmed Snoun.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Snoun, A., Bouchrika, T. & Jemai, O. Deep-learning-based human activity recognition for Alzheimer’s patients’ daily life activities assistance. Neural Comput & Applic 35, 1777–1802 (2023). https://doi.org/10.1007/s00521-022-07883-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07883-1

Keywords

Search

Navigation