Skip to main content
SpringerLink
Log in

Patient activity recognition using radar sensors and machine learning

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Indoor human activity recognition is actively studied as part of creating various intelligent systems with applications in smart home and office, smart health, internet of things, etc. Intrusive devices such as video cameras or sensors attached to the human body are often used to realize human activity recognition. These solutions, however, lead to various privacy issues. On the other hand, radar sensors are privacy-preserving and provide a lot of information about the subject such as speed, distance, range, and angle. Moreover, radar sensors can sense through the walls. In this respect, we investigate the use of radar data to achieve patient activity recognition. In particular, human activity data are collected from both an indoor environment that replicates a hospital setting and a real-life hospital room using two high dimensional radar sensors. The data are further fed to various supervised Machine Learning (ML) classification approaches. We investigate the robustness and generalization capabilities of the ML approaches with respect to people’s age, radar sensor position, mobility aids and environments. The results show promising levels of accuracy. The Convolutional Neural Network (CNN) using Micro-Doppler (MD) maps are more effective for generalizing across different environments and radar positions with 62% and 73% accuracy, respectively. The CNNs using Range-Doppler (RD) maps are more efficient than using MD maps within the same environment in the case of distribution of age (87–95%), mobility aids (91–95%) and with different subjects (93–95%). A subset of the data set is made publicly available.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The data set is publicly available at: https://www.imec-int.com/en/PARrad.

  2. https://www.ugent.be/ea/idlab/en/research/research-infrastructure/homelab.htm.

  3. https://pytorch.org.

  4. https://scikit-learn.org.

References

  1. Ageing and Health (2020) https://www.who.int/news-room/fact-sheets/detail/ageing-and-health

  2. Zhao M, Li T, Abu Alsheikh M, Tian Y, Zhao H, Torralba A, Katabi D (2018) Through-wall human pose estimation using radio signals. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, Salt Lake City, UT, pp 7356–7365

  3. Brooker GM (2005) Understanding millimetre wave FMCW radars. In: 1 St international conference on sensing technology. IEEE, New Zealand, pp 152–157

  4. Iovescu C, Rao S (2017) The fundamentals of millimeter wave. p 9

  5. Chen VC, Li F, Ho SS, Wechsler H (2006) Micro-Doppler effect in radar: phenomenon, model, and simulation study. IEEE Trans Aerosp Electron Syst 42(1):2–21

    Article  Google Scholar 

  6. Hong-Bo Zhang, Yi-Xiang Zhang, Bineng Zhong, Qing Lei, Lijie Yang, Ji-Xiang Du, Duan-Sheng Chen (2019) A comprehensive survey of vision-based human action recognition methods. Sensors 19:1005

    Article  Google Scholar 

  7. Polfliet V, Knudde N, Vandersmissen B, Couckuyt I, Dhaene T (2018) Structured inference networks using high-dimensional sensors for surveillance purposes. In: Engineering applications of neural networks. Springer International Publishing, pp 1–12

  8. Jain DK, Mahanti A, Shamsolmoali P, Manikandan R (2020) Deep neural learning techniques with long short-term memory for gesture recognition. Neural Comput Appl, March

  9. Gutoski M, Lazzaretti AE, Lopes HS (2020) Deep metric learning for open-set human action recognition in videos. Neural Comput Appl, June

  10. Chao Jing, Ping Wei, Hongbin Sun, Nanning Zheng (2020) Spatiotemporal neural networks for action recognition based on joint loss. Neural Comput Appl 32(9):4293–4302

    Article  Google Scholar 

  11. Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks. In: Proceedings of the 2014 IEEE conference on computer vision and pattern recognition, CVPR ’14. IEEE Computer Society, pp 1725–1732

  12. Yue-Hei Ng J, Hausknecht M, Vijayanarasimhan S, Vinyals O, Monga R, Toderici G (2015) Beyond short snippets: deep networks for video classification. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4694–4702, June

  13. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3D convolutional networks. In: The IEEE international conference on computer vision (ICCV), December

  14. Zhang Z, Ma X, Song R, Rong X, Tian X, Tian G, Li Y (2017) Deep learning based human action recognition: a survey. In: 2017 Chinese automation congress (CAC), pp 3780–3785, October

  15. Herath S, Harandi M, Porikli F (2017) Going deeper into action recognition. Image Vision Comput 60(1):4–21

    Article  Google Scholar 

  16. Castro FM, Marín-Jiménez MJ, Guil N, de la Blanca NP (2020) Multimodal feature fusion for CNN-based gait recognition: an empirical comparison. Neural Comput Appl 32(17):14173–14193

    Article  Google Scholar 

  17. Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. In: The IEEE conference on computer vision and pattern recognition (CVPR), June

  18. Tsinganos P, Cornelis B, Cornelis J, Jansen B, Skodras A (2020) Hilbert sEMG data scanning for hand gesture recognition based on deep learning. Neural Comput Appl, July

  19. Singh T, Vishwakarma DK (2020) A deeply coupled ConvNet for human activity recognition using dynamic and RGB images. Neural Comput Appl, May

  20. Falls (2018) https://www.who.int/news-room/fact-sheets/detail/falls

  21. Lau SL, König I, David K, Parandian B, Carius-Düssel C, Schultz M (2010) Supporting patient monitoring using activity recognition with a smartphone. In: 2010 7th international symposium on wireless communication systems, pp 810–814, September

  22. Ichwana D, Arief M, Puteri N, Ekariani S (2018) Movements monitoring and falling detection systems for transient ischemic attack patients using accelerometer based on internet of things. In: 2018 international conference on information technology systems and innovation (ICITSI), pp 491–496, October

  23. Schrader L, Vargas Toro A, Konietzny S, Rüping S, Schäpers B, Steinböck M, Krewer C, Müller F, Güttler J, Bock T (2020) Advanced sensing and human activity recognition in early intervention and rehabilitation of elderly people. J Popul Ageing 13(2):139–165

    Article  Google Scholar 

  24. SA Shah, D Fan, A Ren, N Zhao, X Yang, SAK Tanoli (2018) Seizure episodes detection via smart medical sensing system. J Ambient Intell Hum Comput, November

  25. Khan Muhammad Bilal, Yang Xiaodong, Ren Aifeng, Al-Hababi Mohammed Ali Mohammed, Zhao Nan, Guan Lei, Fan Dou, Shah Syed Aziz (2019) Design of software defined radios based platform for activity recognition. IEEE Access 7:31083–31088

    Article  Google Scholar 

  26. Biagetti G, Crippa P, Falaschetti L, Orcioni S, Turchetti Cl (2018) Human activity monitoring system based on wearable sEMG and accelerometer wireless sensor nodes. BioMed Eng Online 17(Suppl 1):1–18

    Google Scholar 

  27. Georgakopoulos SV, Tasoulis SK, Mallis GI, Vrahatis AG, Plagianakos VP, Maglogiannis IG (2020) Change detection and convolution neural networks for fall recognition. Neural Comput Appl, July

  28. Aimilia Papagiannaki, Zacharaki Evangelia I, Gerasimos Kalouris, Spyridon Kalogiannis, Konstantinos Deltouzos, John Ellul, Vasileios Megalooikonomou (2019) Recognizing physical activity of older people from wearable sensors and inconsistent data. Sensors 19(4):880

    Article  Google Scholar 

  29. Ann-Kathrin Seifert, Amin Moeness G, Zoubir Abdelhak M (2019) Toward unobtrusive in-home gait analysis based on radar Micro-Doppler signatures. IEEE Trans Biomed Eng 66(9):2629–2640

    Article  Google Scholar 

  30. Zhu S, Xu J, Guo H, Liu Q, Wu S, Wang H (2018) Indoor human activity recognition based on ambient radar with signal processing and machine learning. In: 2018 IEEE international conference on communications (ICC), pp 1–6, May

  31. Yang S, Le Kernec J, Fioranelli F, Romain O (2019) Human activities classification in a complex space using raw radar data. In: 2019 international radar conference (RADAR), pp 1–4, September

  32. Linda Senigagliesi, Gianluca Ciattaglia, Adelmo De Santis, Ennio Gambi (2020) People walking classification using automotive radar. Electronics 9(4):588

    Article  Google Scholar 

  33. Sevgi Gurbuz, Moeness Amin (2019) Radar-based human-motion recognition with deep learning: promising applications for indoor monitoring. IEEE Signal Process Mag 36:16–28

    Google Scholar 

  34. Hao Du, Tian Jin, Yuan He, Yongping Song, Yongpeng Dai (2020) Segmented convolutional gated recurrent neural networks for human activity recognition in ultra-wideband radar. Neurocomputing 396:451–464

    Article  Google Scholar 

  35. Lien J, Gillian N, Karagozler M, Amihood P, Schwesig C, Olson E, Raja H, Poupyrev I (2016) Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Trans Graph 35:1–19

    Article  Google Scholar 

  36. Wang S, Song J, Lien J, Poupyrev I, Hilliges O (2016) Interacting with soli: exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In: Proceedings of the 29th annual symposium on user interface software and technology, UIST ’16. ACM, pp 851–860

  37. Vandersmissen B, Knudde N, Jalalvand A, Couckuyt I, Dhaene T, De Neve W (2019) Indoor human activity recognition using high-dimensional sensors and deep neural networks. Neural Comput Appl, August

  38. Fioranelli F, Shah SA, Li H, Shrestha A, Yang S, Le Kernec J (2019) Radar sensing for healthcare. Electron Lett 55(19):1022–1024

    Article  Google Scholar 

  39. Chuanwei Ding, Hong Hong Yu, Chu Zou Hui, Xiaohua Zhu, Francesco Fioranelli, Julien Le Kernec, Changzhi Li (2019) Continuous human motion recognition with a dynamic range-Doppler trajectory method based on FMCW radar. IEEE Trans Geosci Remote Sens 57(9):6821–6831

    Article  Google Scholar 

  40. Zhao H, Hong H, Miao D, Li Y, Zhang H, Zhang Y, Li C, Zhu X (2019) A noncontact breathing disorder recognition system using 2.4-GHz digital-IF Doppler radar. IEEE J Biomed Health Inform 23(1):208–217

    Article  Google Scholar 

  41. Fioranelli F, Le Kernec J, Shah SA (2019) Radar for health care: recognizing human activities and monitoring vital signs. IEEE Potent 38(4):16–23

    Article  Google Scholar 

  42. Shah SA, Fioranelli F (2019) Human activity recognition: preliminary results for dataset portability using FMCW radar. In: 2019 international radar conference (RADAR), pp 1–4, September

  43. Gurbuz SZ, Clemente C, Balleri A, Soraghan JJ (2017) Micro-Doppler-based in-home aided and unaided walking recognition with multiple radar and sonar systems. IET Radar Sonar Navigat 11(1):107–115

    Article  Google Scholar 

  44. Haobo Li, Aman Shrestha, Hadi Heidari, Julien Le Kernec, Francesco Fioranelli (2020) Bi-LSTM network for multimodal continuous human activity recognition and fall detection. IEEE Sens J 20(3):1191–1201

    Article  Google Scholar 

  45. Branka Jokanović, Moeness Amin (2018) Fall detection using deep learning in range-Doppler radars. IEEE Trans Aerosp Electron Syst 54(1):180–189

    Article  Google Scholar 

  46. Haobo Li, Aman Shrestha, Hadi Heidari, Julien Le Kernec, Francesco Fioranelli (2019) Magnetic and radar sensing for multimodal remote health monitoring. IEEE Sens J 19(20):8979–8989

    Article  Google Scholar 

  47. Jia M, Li S, Le Kernec J, Yang S, Fioranelli F, Romain O (2020) Human activity classification with radar signal processing and machine learning. In: 2020 International conference on UK-China emerging technologies (UCET), pp 1–5, August

  48. Fioranelli F, Shah SA, Li H, Shrestha A, Yang S, Le Kernec J (2019) Radar signatures of human activities. http://researchdata.gla.ac.uk/848/. https://doi.org/10.5525/gla.researchdata.848, July

  49. Ritchie M, Capraru R, Fioranelli F (2020) Dop-NET: a micro-Doppler radar data challenge. Electron Lett 56, February

  50. Baptist Vandersmissen, Nicolas Knudde, Azarakhsh Jalalvand, Ivo Couckuyt, André Bourdoux, Wesley De Neve, Tom Dhaene (2018) Indoor person identification using a low-power FMCW radar. IEEE Trans Geosci Remote Sens 56(7):3941–3952

    Article  Google Scholar 

  51. Jri Lee, Yi-An Li, Meng-Hsiung Hung, Shih-Jou Huang (2010) A fully-integrated 77-GHz FMCW radar transceiver in 65-nm CMOS technology. IEEE J Solid State Circuits 45(12):2746–2756

    Article  Google Scholar 

  52. Chen Q, Tan B, Chetty K, Woodbridge K (2016) Activity recognition based on micro-Doppler signature with in-home Wi-Fi. In: 2016 IEEE 18th international conference on E-Health networking, applications and services (Healthcom), pp 1–6, September

  53. Chang C-C, Lin C-J (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):27:1-27:27

    Article  Google Scholar 

  54. Leo Breiman (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  55. Alex Krizhevsky, Ilya Sutskever, Hinton Geoffrey E (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  56. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: Fürnkranz J, Joachims T (eds) Proceedings of the 27th international conference on machine learning (ICML-10). Omnipress, pp 807–814

  57. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp 315–323, June

  58. Clevert D-A, Unterthiner T, Hochreiter S (2016) Fast and accurate deep network learning by exponential linear units (ELUs). arXiv:1511.07289 [cs], February

Download references

Acknowledgements

The research activities described in this paper were funded by Ghent University-imec and the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

Author information

Authors and Affiliations

  1. Department of Information Technology (INTEC), Internet Technology and Data Science Lab (IDLab) Research Group,iGent, Ghent University–imec, Technologiepark-Zwijnaarde 126, 9052, Ghent, Belgium

    Geethika Bhavanasi, Lorin Werthen-Brabants, Tom Dhaene & Ivo Couckuyt

Authors
  1. Geethika Bhavanasi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorin Werthen-Brabants
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tom Dhaene
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ivo Couckuyt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Geethika Bhavanasi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: Physical characteristics of the participants involved in Homelab and Hospital data sets.

Appendix: Physical characteristics of the participants involved in Homelab and Hospital data sets.

See the Tables 10 and 11.

Table 10 Homelab data set: physical characteristics of the participants
Full size table
Table 11 Hospital data set: physical characteristics of the participants. The Elderly people who were not able to perform ‘fall on the floor’ and ‘stand up from the floor’ activities are indicated with ‘*’ sign in the gender group
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bhavanasi, G., Werthen-Brabants, L., Dhaene, T. et al. Patient activity recognition using radar sensors and machine learning. Neural Comput & Applic 34, 16033–16048 (2022). https://doi.org/10.1007/s00521-022-07229-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07229-x

Keywords

Search

Navigation