Skip to main content
SpringerLink
Log in

Personalized models for human activity recognition with wearable sensors: deep neural networks and signal processing

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Human Activity Recognition (HAR) has been attracting research attention because of its importance in applications such as health monitoring, assisted living, and active living. In recent years, deep learning, specifically Convolutional Neural Networks (CNNs), have been achieving great results due to their ability to extract features and model complex actions. These generic models work great for the subjects on which they were trained, but their performance degrades substantially for new subjects. Consequently, this paper proposes a personalized HAR model based on CNN and signal decomposition. First, features are extracted from multi-modal sensor data with signal processing techniques, including Stationary Wavelet Transform, Empirical Mode Decomposition (EMD), and Ensemble EMD. Next, CNN carries out the information fusion and the final classification. Personalization is achieved by using a few seconds of the target subject data to select the version of the trained CNN best suited for the target subject. Results show that EMD with cubic spline achieves better accuracy than other signal processing techniques. Moreover, the proposed approach, irrelevant of the type of signal processing, outperforms the state-of-the-art CNN approaches with time-domain features.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

MHEALTH [54] dataset is available from http://archive.ics.uci.edu/ml/datasets/mhealth+dataset.

WISDM [58] dataset is available from: https://archive.ics.uci.edu/ml/datasets/WISDM+Smartphone+and+Smartwatch+Activity+and+Biometrics+Dataset+https://archive.ics.uci.edu/ml/datasets/WISDM+Smartphone+and+Smartwatch+Activity+and+Biometrics+Dataset+https://archive.ics.uci.edu/ml/datasets/WISDM+Smartphone+and+Smartwatch+Activity+and+Biometrics+Dataset+.

References

  1. Grossi G, Lanzarotti R, Napoletano P, Noceti N, Odone F (2020) Positive technology for elderly well-being: a review. Pattern Recogn Lett 137:61–70

    Article  Google Scholar 

  2. Li Q, Gravina R, Li Y, Alsamhi SH, Sun F, Fortino G (2020) Multi-user activity recognition challenges and opportunities. Inform Fus 63:121–135

    Article  Google Scholar 

  3. Nweke HF, Teh YW, Mujtaba G, Al-Garadi MA (2019) Data fusion and multiple classifier systems for human activity detection and health monitoring: review and open research directions. Inform Fus 46:147–170

    Article  Google Scholar 

  4. Mohsen S, Elkaseer A, Scholz SG (2021) Human activity recognition using k-nearest neighbor machine learning algorithm. In: Proceedings of the international conference on sustainable design and manufacturing. Springer, pp 304–313

  5. Ahmed N, Rafiq JI, Islam MR (2020) Enhanced human activity recognition based on smartphone sensor data using hybrid feature selection model. Sensors, 20(1)

  6. Hu C, Chen Y, Hu L, Peng X (2018) A novel random forests based class incremental learning method for activity recognition. Pattern Recogn 78:277–290

    Article  Google Scholar 

  7. Asghari P, Soleimani E, Nazerfard E (2020) Online human activity recognition employing hierarchical hidden Markov models. J Ambient Intell Human Comput 11(3):1141–1152

    Article  Google Scholar 

  8. Chen K, Zhang D, Yao L, Guo B, Yu Z, Liu Y (2021) Deep learning for sensor-based human activity recognition overview, challenges, and opportunities. ACM Comput Surv (CSUR) 54(4):1–40

    Google Scholar 

  9. Gandarias JM, Garcia-Cerezo AJ, Gomez-De Gabriel JM (2019) CNN-Based methods for object recognition with high-resolution tactile sensors. IEEE Sensors J 19(16):6872–6882

    Article  Google Scholar 

  10. Batbaatar E, Li M, Ryu KH (2019) Semantic-emotion neural network for emotion recognition from text. IEEE Access 7:111866–111878

    Article  Google Scholar 

  11. Asghar MZ, Habib A, Habib A, Khan A, Ali R, Khattak A (2021) Exploring deep neural networks for rumor detection. J Ambient Intell Human Comput 12(4):4315–4333

    Article  Google Scholar 

  12. Rai K, Hojatpanah F, Badrkhani Ajaei F, Grolinger K (2021) Deep learning for high-impedance fault detection: Convolutional autoencoders. Energies, 14(12)

  13. Hassan MM, Alam MGR, Uddin MZ, Huda S, Almogren A, Fortino G (2019) Human emotion recognition using deep belief network architecture. Inform Fus 51:10–18

    Article  Google Scholar 

  14. Huang Y, Tian K, Wu A, Zhang G (2019) Feature fusion methods research based on deep belief networks for speech emotion recognition under noise condition. J Ambient Intell Human Comput 10 (5):1787–1798

    Article  Google Scholar 

  15. Ghosh AM, Grolinger K (2020) Edge-cloud computing for internet of things data analytics embedding intelligence in the edge with deep learning. IEEE Trans Industr Inform 17(3):2191–2200

    Google Scholar 

  16. Gjoreski M, Janko V , Slapničar G , Mlakar M, Reščič N, Bizjak J, Drobnič V, Marinko M, Mlakar N, Lustreck M et al (2020) Classical and deep learning methods for recognizing human activities and modes of transportation with smartphone sensors. Inform Fus 62:47–62

    Article  Google Scholar 

  17. Qin Z, Zhang Y, Meng S, Qin Z, Choo KKR (2020) Imaging and fusing time series for wearable sensor-based human activity recognition. Inform Fus 53:80–87

    Article  Google Scholar 

  18. Mohammad Y, Matsumoto K, Hoashi K (2018) Primitive activity recognition from short sequences of sensory data. Appl Intell 48(10):3748–3761

    Article  Google Scholar 

  19. Ramasamy Ramamurthy S, Roy N (2018) Recent trends in machine learning for human activity recognition—a survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4)

  20. He H, Tan Y, Zhang W (2018) A wavelet tensor fuzzy clustering scheme for multi-sensor human activity recognition. Eng Appl Artif Intell 70:109–122

    Article  Google Scholar 

  21. Dang LM, Min K, Wang H, Piran MJ, Lee CH, Moon H (2020) Sensor-based and vision-based human activity recognition: A comprehensive survey. Pattern Recogn, 108

  22. Hashim M, Amutha R (2021) Human activity recognition based on smartphone using fast feature dimensionality reduction technique. J Ambient Intell Human Comput 12:2365–2374

    Article  Google Scholar 

  23. Gravina R, Alinia P, Ghasemzadeh H, Giancarlo F (2017) Multi-sensor fusion in body sensor networks state-of-the-art and research challenges. Inform Fus 35:68–80

    Article  Google Scholar 

  24. Chen S, Liu P, Tang D, Tao S, Zhang T (2021) Identification of thin-layer coal texture using geophysical logging data Investigation by wavelet transform and linear discrimination analysis. Int J Coal Geol, 239

  25. Zhang D (2019) Wavelet transform. In: Fundamentals of image data mining. Springer, pp 35–44

  26. Pandit S, Sharma S (2022) On the use of wavelets for analysis of nanofluid flow and thermal transmission through asymmetric porous channel. In: Proceedings of the National Academy of Sciences, India Section A: Physical Sciences, pp 1–13

  27. Nolasco DHS, Costa FB , Palmeira ES, Alves DK, Bedregal BR, Rocha TO, Ribeiro RL, Silva JC (2019) Wavelet-fuzzy power quality diagnosis system with inference method based on overlap functions: case study in an ac microgrid. Eng Appl Artif Intell 85:284–294

    Article  Google Scholar 

  28. Abdu-Aguye MG, Gomaa W (2019) Competitive feature extraction for activity recognition based on wavelet transforms and adaptive pooling. In: IEEE International joint conference on neural networks

  29. Wu Z, Huang NE (2009) Ensemble empirical mode decomposition: a noise-assisted data analysis method. Advances in Adaptive Data Analysis 1(01):1–41

    Article  Google Scholar 

  30. Fairchild DP, Narayanan RM (2014) Classification of human motions using empirical mode decomposition of human micro-doppler signatures. IET Radar, Sonar & Navigation 8(5):425– 434

    Article  Google Scholar 

  31. Tian Y, Zhang J, Wang J, Geng Y, Wang X (2020) Robust human activity recognition using single accelerometer via wavelet energy spectrum features and ensemble feature selection. Syst Sci Control Eng 8(1):83–96

    Article  Google Scholar 

  32. Xu H, Liu J, Hu H, Zhang Y (2016) Wearable sensor-based human activity recognition method with multi-features extracted from hilbert-huang transform. Sensors, 16(12)

  33. Gholamiangonabadi D, Kiselov N, Grolinger K (2020) Deep neural networks for human activity recognition with wearable sensors leave-one-subject-out cross-validation for model selection. IEEE Access 8:133982–133994

    Article  Google Scholar 

  34. Daubechies I (1992) Ten lectures on wavelets society for industrial and applied mathematics

  35. Stéphane M (1999) A wavelet tour of signal processing. Elsevier

  36. Ray PK, Panigrahi BK, Rout PK, Mohanty Asit, Dubey H (2017) Detection of faults in power system using wavelet transform and independent component analysis. In: Computer communication and electrical technology: proceedings of the international conference on advancement of computer communication and electrical technology

  37. Omerhodzic I, Avdakovic S, Nuhanovic A, Dizdarevic K (2010) Energy distribution of eeg signals: eeg signal wavelet-neural network classifier. Int J Biomed Biol Eng 4(1)

  38. Eneyew D, Capretz MA, Bitsuamlak G (2020) London hydro predicting residential energy consumption using wavelet decomposition with deep neural network

  39. Gong M, Wang J, Bai Y, Li B, Zhang L (2020) Heatload prediction of residential buildings based on discrete wavelet transform and tree-based ensemble learning. J Building Eng 32

  40. Azzaoui H, Mansouri I, Elkihel B (2019) Methylcyclohexane continuous distillation column fault detection using stationary wavelet transform & fuzzy c-means. Materials Today: Proceedings, vol 13

  41. Huang NE, Shen Z, Long SR, Wu MC, Shih HH, Zheng Q, Yen N-C, Tung CC, Liu HH (1998) The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proceedings of the Royal Society of London. Series A: Math Phys Eng Sci 454(1971):903–995

    Article  MathSciNet  MATH  Google Scholar 

  42. Liu Y, Pu H, Sun D-W (2021) Efficient extraction of deep image features using convolutional neural network (cnn) for applications in detecting and analysing complex food matrices. Trends Food Sci Technol 113:193–204

    Article  Google Scholar 

  43. Mario M-O (2018) Human activity recognition based on single sensor square hv acceleration images and convolutional neural networks. IEEE Sensors J 19(4):1487–1498

    Article  Google Scholar 

  44. Kim SY, Han HG, Kim JW, Lee S, Kim TW (2017) A hand gesture recognition sensor using reflected impulses. IEEE Sensors J 17(10):2975–2976

    Article  Google Scholar 

  45. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press

  46. Ha S, Yun J-M, Choi S (2015) Multi-modal convolutional neural networks for activity recognition. In: IEEE International conference on systems, man, and cybernetics

  47. Md ZU, Hassan MM (2018) Activity recognition for cognitive assistance using body sensors data and deep convolutional neural network. IEEE Sensors J 19(19):8413–8419

    Google Scholar 

  48. Hanai Y, Nishimura J, Kuroda T (2009) Haar-like filtering for human activity recognition using 3d accelerometer. In: IEEE 13Th digital signal processing workshop and 5th IEEE signal processing education workshop

  49. He Z, Jin L (2009) Activity recognition from acceleration data based on discrete consine transform and SVM. In: IEEE international conference on systems, man and cybernetics

  50. Assam R, Seidl T (2014) Activity recognition from sensors using dyadic wavelets and hidden Markov model. In: IEEE 10th international conference on wireless and mobile computing, networking and communications

  51. Ferrari A, Micucci D, Mobilio M, Napoletano P (2020) On the personalization of classification models for human activity recognition. IEEE Access 8:32066–32079

    Article  Google Scholar 

  52. Vu TT, Fujinami K (2019) Understanding compatibility-based classifier personalization in activity recognition. In: Joint 8th IEEE international conference on informatics, electronics & vision and 3rd international conference on imaging, vision & pattern recognition

  53. Lin C-Y, Marculescu R (2020) Model personalization for human activity recognition. In: IEEE international conference on pervasive computing and communications workshops

  54. Banos O, Garcia R, Holgado-Terriza JA, Damas M, Pomares H, Rojas I, Saez A, Villalonga C (2014) MHealthDroid: a novel framework for agile development of mobile health applications. In: International workshop on ambient assisted living

  55. Banos O, Villalonga C, Garcia R, Saez A, Damas M, Holgado-Terriza JA, Lee S, Pomares H, Rojas I (2015) Design, implementation and validation of a novel open framework for agile development of mobile health applications. Biomed Eng Online 14(2):1–20

    Google Scholar 

  56. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  57. Khatun S, Morshed BI (2018) Fully-automated human activity recognition with transition awareness from wearable sensor data for mhealth. In: 2018 IEEE international conference on electro/information technology, pp 0934–0938. IEEE

  58. Gary MW, Yoneda K, Hayajneh T (2019) Smartphone and smartwatch-based biometrics using activities of daily living. IEEE Access 7:133190–133202

    Article  Google Scholar 

  59. Rokni SA, Nourollahi M, Ghasemzadeh H (2018) Personalized human activity recognition using convolutional neural networks. In: Proceedings of the AAAI conference on artificial intelligence, vol 32

  60. Bragança H, Colonna JG, Lima WS, Souto E (2020) A smartphone lightweight method for human activity recognition based on information theory. Sensors, 20(7)

  61. Jordao A, Nazare Jr AC, Sena J, Schwartz WR (2019) Human activity recognition based on wearable sensor data: a standardization of the state-of-the-art

Download references

Acknowledgements

This work was supported by Natural Sciences and Engineering Research Council of Canada under Grant RGPIN-2018-06222.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Western University, London, ON, N6A 5B9, Canada

    Davoud Gholamiangonabadi & Katarina Grolinger

Authors
  1. Davoud Gholamiangonabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Katarina Grolinger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katarina Grolinger.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gholamiangonabadi, D., Grolinger, K. Personalized models for human activity recognition with wearable sensors: deep neural networks and signal processing. Appl Intell 53, 6041–6061 (2023). https://doi.org/10.1007/s10489-022-03832-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03832-6

Keywords

Search

Navigation