Skip to main content
Springer Nature Link
Log in

Multi-input CNN-GRU based human activity recognition using wearable sensors

  • Regular Paper
  • Published:
Computing Aims and scope Submit manuscript

Abstract

Human Activity Recognition (HAR) has attracted much attention from researchers in the recent past. The intensification of research into HAR lies in the motive to understand human behaviour and inherently anticipate human intentions. Human activity data obtained via wearable sensors like gyroscope and accelerometer is in the form of time series data, as each reading has a timestamp associated with it. For HAR, it is important to extract the relevant temporal features from raw sensor data. Most of the approaches for HAR involves a good amount of feature engineering and data pre-processing, which in turn requires domain expertise. Such approaches are time-consuming and are application-specific. In this work, a Deep Neural Network based model, which uses Convolutional Neural Network, and Gated Recurrent Unit is proposed as an end-to-end model performing automatic feature extraction and classification of the activities as well. The experiments in this work were carried out using the raw data obtained from wearable sensors with nominal pre-processing and don’t involve any handcrafted feature extraction techniques. The accuracies obtained on UCI-HAR, WISDM, and PAMAP2 datasets are 96.20%, 97.21%, and 95.27% respectively. The results of the experiments establish that the proposed model achieved superior classification performance than other similar architectures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Khan ZA, Sohn W (2013) A hierarchical abnormal human activity recognition system based on R-transform and kernel discriminant analysis for elderly health care. Computing 95:109–127. https://doi.org/10.1007/s00607-012-0216-x

    Article  Google Scholar 

  2. Cornacchia M, Ozcan K, Zheng Y, Velipasalar S (2016) A survey on activity detection and classification using wearable sensors. IEEE Sens J 17:386–403

    Article  Google Scholar 

  3. Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2013) A public domain dataset for human activity recognition using smartphones. In: ESANN

  4. Feng Z, Mo L, Li M (2015) A Random Forest-based ensemble method for activity recognition. In: 37th Annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 5074–5077

  5. Jain A, Kanhangad V (2017) Human activity classification in smartphones using accelerometer and gyroscope sensors. IEEE Sens J 18(3):1169–1177

    Article  Google Scholar 

  6. Wang Z, Wu D, Chen J, Ghoneim A, Hossain MA (2016) A triaxial accelerometer-based human activity recognition via EEMD-based features and game-theory-based feature selection. IEEE Sens J. 16(9):3198–3207

    Article  Google Scholar 

  7. Guo M, Wang Z, Yang N, Li Z, An T (2018) A multisensor multiclassifier hierarchical fusion model based on entropy weight for human activity recognition using wearable inertial sensors. IEEE Trans Hum–Mach Syst. 49(1):105–111

    Article  Google Scholar 

  8. Nweke HF, Teh YW, Al-Garadi MA, Alo UR (2018) Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Syst Appl 105:233–261

    Article  Google Scholar 

  9. Zhang X, Zhang Y, Hu Q (2019) Deep learning based vein segmentation from susceptibility-weighted images. Computing 101(6):637–652. https://doi.org/10.1007/s00607-018-0677-7

    Article  Google Scholar 

  10. Yu X, Dong H (2018) PTL-CFS based deep convolutional neural network model for remote sensing classification. Computing 100(8):773–785. https://doi.org/10.1007/s00607-018-0609-6

    Article  Google Scholar 

  11. Semwal VB, Mondal K, Nandi GC (2017) Robust and accurate feature selection for humanoid push recovery and classification: deep learning approach. Neural Comput Appl 28:565–574

    Article  Google Scholar 

  12. Ren S, He K, Girshick R, Sun J(2015) Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint http://arxiv.org/abs/1506.01497

  13. Al-Makhadmeh Z, Tolba A (2020) Automatic hate speech detection using killer natural language processing optimizing ensemble deep learning approach. Computing 102(2):501–522. https://doi.org/10.1007/s00607-019-00745-0

    Article  Google Scholar 

  14. Le Cun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Handwritten digit recognition with a back-propagation network. In: Proceedings of the 2nd international conference on neural information processing systems, pp 396–404

  15. Lee H, Grosse R, Ranganath R, Ng AY (2009) Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of the 26th annual international conference on machine learning, pp 609–616

  16. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint http://arxiv.org/abs/1412.3555

  17. Wan S, Qi L, Xu X, Tong C, Gu Z (2020) Deep learning models for real-time human activity recognition with smartphones. Mobile Networks Appl 25(2):743–755

    Article  Google Scholar 

  18. Liu CL, Hsaio WH, Tu YC (2018) Time series classification with multivariate convolutional neural network. IEEE Trans Ind Electron 66(6):4788–4797

    Article  Google Scholar 

  19. Cui Z, Chen W, Chen Y (2016) Multi-scale convolutional neural networks for time series classification. arXiv preprint http://arxiv.org/abs/1603.06995

  20. Ignatov A (2018) Real-time human activity recognition from accelerometer data using Convolutional Neural Networks. Appl Soft Comput 62:915–922

    Article  Google Scholar 

  21. Ronao CA, Cho SB (2016) Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244

    Article  Google Scholar 

  22. Chen Y, Zhong K, Zhang J, Sun Q, Zhao X (2016) LSTM networks for mobile human activity recognition. In: 2016 International conference on artificial intelligence: technologies and applications pp 50–53

  23. Kwapisz JR, Weiss GM, Moore SA (2011) Activity recognition using cell phone accelerometers. ACM SIGKDD Explor Newsl 12(2):74–82

    Article  Google Scholar 

  24. Ullah M, Ullah H, Khan SD, Cheikh FA (2019) Stacked lstm network for human activity recognition using smartphone data. In: 8th European workshop on visual information processing (EUVIP). IEEE, pp 175–180

  25. Yu S, Qin L (2018) Human activity recognition with smartphone inertial sensors using bidir-lstm networks. In: 3rd international conference on mechanical, control and computer engineering (ICMCCE). IEEE, pp 219–224

  26. Zhao Y, Yang R, Chevalier G, Xu X, Zhang Z (2018) Deep residual bidir-LSTM for human activity recognition using wearable sensors. Math Problems Eng. https://doi.org/10.1155/2018/7316954

    Article  Google Scholar 

  27. Xu C, Chai D, He J, Zhang X, Duan S (2019) InnoHAR: A deep neural network for complex human activity recognition. IEEE Access 7:9893–9902

    Article  Google Scholar 

  28. Mutegeki R, Han DS (2020) A CNN-LSTM approach to human activity recognition. In: International conference on artificial intelligence in information and communication (ICAIIC). IEEE, pp 362–366

  29. Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866

    Article  Google Scholar 

  30. Canizo M, Triguero I, Conde A, Onieva E (2019) Multi-head CNN–RNN for multi-time series anomaly detection: An industrial case study. Neurocomputing 363:246–260

    Article  Google Scholar 

  31. Karim F, Majumdar S, Darabi H, Chen S (2017) LSTM fully convolutional networks for time series classification. IEEE access 6:1662–1669

    Article  Google Scholar 

  32. Kaushik S, Choudhury A, Dasgupta N, Natarajan S, Pickett LA, Dutt V (2020) Ensemble of multi-headed machine learning architectures for time-series forecasting of healthcare expenditures. Applications of Machine Learning. Springer, Singapore, pp 199–216

    Chapter  Google Scholar 

  33. Patil P, Kumar KS, Gaud N, Semwal VB (2019) Clinical human gait classification: extreme learning machine approach. In: 1st International conference on advances in science, engineering and robotics technology (ICASERT). IEEE, pp 1–6

  34. Semwal VB, Gaud N, Nandi GC (2019) Human gait state prediction using cellular automata and classification using ELM. Machine intelligence and signal analysis. Springer, Singapore, pp 135–145

    Chapter  Google Scholar 

  35. Wang Z, Wu D, Gravina R, Fortino G, Jiang Y, Tang K (2017) Kernel fusion based extreme learning machine for cross-location activity recognition. Information Fusion 37:1–9

    Article  Google Scholar 

  36. Quaid MA, Jalal A (2020) Wearable sensors based human behavioral pattern recognition using statistical features and reweighted genetic algorithm. Multimed Tools Appl 79(9):6061–6083

    Article  Google Scholar 

  37. Zhang Y, Zhang Z, Zhang Y, Bao J, Zhang Y, Deng H (2019) Human activity recognition based on motion sensor using u-net. IEEE Access 7:75213–75226

    Article  Google Scholar 

  38. Lu J, Zheng X, Sheng M, Jin J, Yu S (2020) Efficient human activity recognition using a single wearable sensor. IEEE Internet Things J 7(11):11137–11146

    Article  Google Scholar 

  39. Reiss A, Stricker D (2012) Introducing a new benchmarked dataset for activity monitoring. In: 16th International symposium on wearable computers. IEEE, pp 108–109

  40. Ignatov AD, Strijov VV (2016) Human activity recognition using quasiperiodic time series collected from a single tri-axial accelerometer. Multimed Tools Appl 75(12):7257–7270

    Article  Google Scholar 

  41. Sadouk L. CNN approaches for time series classification (2019) In: Time series analysis-data, methods, and applications. IntechOpen, pp 1–23

  42. Semwal VB, Nandi GC (2016) Generation of joint trajectories using hybrid automate-based model: a rocking block-based approach. IEEE Sens J 16(14):5805–5816

    Article  Google Scholar 

  43. Gupta A, Semwal VB (2020) Multiple task human gait analysis and identification: ensemble learning approach. Emotion and Information Processing. Springer, Cham, pp 185–197

    Chapter  Google Scholar 

  44. Raj M, Semwal VB, Nandi GC (2018) Bidirectional association of joint angle trajectories for humanoid locomotion: the restricted Boltzmann machine approach. Neural Comput Appl 30(6):1747–1755

    Article  Google Scholar 

  45. Nandi GC, Semwal VB, Raj M, Jindal A (2016) Modeling bipedal locomotion trajectories using hybrid automata. In: IEEE region 10 conference (TENCON). IEEE, pp 1013–1018

  46. LeCun Y, Bengio Y (1995) Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361(10)

  47. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Networks 5(2):157–166

    Article  Google Scholar 

  48. Cho K, Van Merriënboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint http://arxiv.org/abs/1409.1259

  49. Hammerla NY, Halloran S, Plötz T (2016) Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv preprint http://arxiv.org/abs/1604.08880

  50. Yang Z, Raymond OI, Zhang C, Wan Y, Long J (2018) DFTerNet: Towards 2-bit dynamic fusion networks for accurate human activity recognition. IEEE Access 6:56750–56764

    Article  Google Scholar 

  51. Cheng X, Zhang L, Tang Y, Liu Y, Wu H, He J (2020) Real-time human activity recognition using conditionally parametrized convolutions on mobile and wearable devices. arXiv preprint http://arxiv.org/abs/2006.03259

  52. Zeng M, Gao H, Yu T, Mengshoel OJ, Langseth H, Lane I, Liu X (2018) Understanding and improving recurrent networks for human activity recognition by continuous attention. In: Proceedings of the 2018 ACM international symposium on wearable computers pp 56–63

Download references

Funding

The author(s) also like to express thanks to SERB, DST, Govt. of India for funding the project under the schema of Early Career Award (ECR), with file no DST No: ECR/2018/000203 dated 04/06/2019.

Author information

Authors and Affiliations

  1. Department of ECE, NIT Jamshedpur, Jamshedpur, Jharkhand, India

    Nidhi Dua & Shiva Nand Singh

  2. Department of CSE, MANIT Bhopal, Bhopal, M.P, India

    Vijay Bhaskar Semwal

Authors
  1. Nidhi Dua
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shiva Nand Singh
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Vijay Bhaskar Semwal
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nidhi Dua.

Ethics declarations

Conflict of interest

We, the authors declare that we have no conflict of interest with any person or organization. This manuscript is based on original research findings done by the authors themselves.

Ethical statement

All the ethical issues have been taken care of while writing the manuscript and we have complied with all the standards to the best of our knowledge.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dua, N., Singh, S.N. & Semwal, V.B. Multi-input CNN-GRU based human activity recognition using wearable sensors. Computing 103, 1461–1478 (2021). https://doi.org/10.1007/s00607-021-00928-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-021-00928-8

Keywords

Mathematics Subject Classification