Skip to main content
SpringerLink
Log in

Enhancing human activity recognition using deep learning and time series augmented data

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Human activity recognition is concerned with detecting different types of human movements and actions using data gathered from various types of sensors. Deep learning approaches, when applied on time series data, offer promising results over intensive handcrafted feature extraction techniques that are highly reliant on the quality of defined domain parameters. In this paper, we investigate the benefits of time series data augmentation in improving the accuracy of several deep learning models on human activity data gathered from mobile phone accelerometers. More specifically, we compare the performance of the Vanilla, Long-Short Term Memory, and Gated Recurrent Units neural network models on three open-source datasets. We use two time series data augmentation techniques and study their impact on the accuracy of the target models. The experiments show that using gated recurrent units achieves the best results in terms of accuracy and training time followed by the long-short term memory technique. Furthermore, the results show that using data augmentation significantly enhances recognition quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Kudlur M (2016) Tensorflow: A system for large-scale machine learning. In: 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pp 265–283

  • Alsarhan T, Alawneh L, Al-Zinati M, Al-Ayyoub M (2019) Bidirectional gated recurrent units for human activity recognition using accelerometer data. In: 2019 IEEE SENSORS, IEEE, pp 1–4

  • Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2013) A public domain dataset for human activity recognition using smartphones. In: Esann, p 3

  • Behera A, Hogg DC, Cohn AG (2012) Egocentric activity monitoring and recovery. Asian conference on computer vision. Springer, Berlin, pp 519–532

    Google Scholar 

  • Bidargaddi N, Sarela A, Klingbeil L, Karunanithi M (2007) Detecting walking activity in cardiac rehabilitation by using accelerometer. In: 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information. IEEE, pp 555–560

  • Che Z, Purushotham S, Cho K, Sontag D, Liu Y (2018) Recurrent neural networks for multivariate time series with missing values. Sci Rep 8(1):1–12

    Google Scholar 

  • Chen Y, Zhong K, Zhang J, Sun Q, Zhao X (2016) Lstm networks for mobile human activity recognition. 2016 International conference on artificial intelligence: technologies and applications. Atlantis Press, Paris

    Google Scholar 

  • Chen Z, Zhu Q, Soh YC, Zhang L (2017) Robust human activity recognition using smartphone sensors via CT-PCA and online SVM. IEEE Trans Industr Inf 13(6):3070–3080

    Article  Google Scholar 

  • Cho H, Yoon SM (2018) Divide and conquer-based 1D CNN human activity recognition using test data sharpening. Sensors 18(4):1055

    Article  Google Scholar 

  • Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Conference on empirical methods in natural language processing (EMNLP 2014), Doha, Qatar, pp 1724–1734

  • Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 workshop on deep learning

  • Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4690–4699

  • Dobbin KK, Simon RM (2011) Optimally splitting cases for training and testing high dimensional classifiers. BMC Med Genom 4(1):31

    Article  Google Scholar 

  • Ferrari A, Micucci D, Mobilio M, Napoletano P (2020) On the personalization of classification models for human activity recognition. IEEE Access 8:32066–32079

    Article  Google Scholar 

  • Foerster F, Smeja M, Fahrenberg J (1999) Detection of posture and motion by accelerometry: a validation study in ambulatory monitoring. Comput Hum Behav 15(5):571–583

    Article  Google Scholar 

  • Gao W, Zhang L, Teng Q, Wu H, Min F, He J (2020) DanHAR: dual attention network for multimodal human activity recognition using wearable sensors.

  • Gers FA, Schraudolph NN, Schmidhuber J (2002) Learning precise timing with LSTM recurrent networks. J Mach Learn Res 3:115–143

    MathSciNet  MATH  Google Scholar 

  • Graves A, Liwicki M, Fernández S, Bertolami R, Bunke H, Schmidhuber J (2008) A novel connectionist system for unconstrained handwriting recognition. IEEE Trans Pattern Anal Mach Intell 31(5):855–868

    Article  Google Scholar 

  • Gutchess D, Checka N, Snorrason MS (2007) Learning patterns of human activity for anomaly detection. Intelligent computing: theory and applications. International Society for Optics and Photonics, Washington, p 65600Y

    Google Scholar 

  • Hammerla NY, Halloran S, Plötz T (2016) Deep, convolutional, and recurrent models for human activity recognition using wearables. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, pp 1533–1540

  • Hassan MM, Uddin MZ, Mohamed A, Almogren A (2018) A robust human activity recognition system using smartphone sensors and deep learning. Future Gener Comput Syst 81:307–313

    Article  Google Scholar 

  • Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  • Husken M, Stagge P (2003) Recurrent neural networks for time series classification. Neurocomputing 50:223–235

    Article  MATH  Google Scholar 

  • Hyndman R, Koehler AB, Ord JK, Snyder RD (2008) Forecasting with exponential smoothing: the state space approach. Springer Science and Business Media, Cham

    Book  MATH  Google Scholar 

  • Jalal A, Kamal S, Kim D (2014) A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors 14(7):11735–11759

    Article  Google Scholar 

  • Jiang W, Yin Z (2015) Human activity recognition using wearable sensors by deep convolutional neural networks. In: Proceedings of the 23rd ACM international conference on Multimedia, pp 1307–1310

  • Johnson RA, Miller I, Freund JE (2000) Probability and statistics for engineers. Pearson Education, London, p 642

    Google Scholar 

  • Jordao A, Kloss R, Schwartz WR (2018) Latent HyperNet: exploring the layers of convolutional neural networks. In: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1–7

  • Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  • Kolekar MH, Dash DP (2016) Hidden markov model based human activity recognition using shape and optical flow based features. In: 2016 IEEE Region 10 Conference (TENCON), IEEE, pp 393–397

  • Kwapisz JR, Weiss GM, Moore SA (2011) Activity recognition using cell phone accelerometers. ACM SIGKDD Explor Newsl 12(2):74–82

    Article  Google Scholar 

  • Li H, Trocan M (2019) Deep learning of smartphone sensor data for personal health assistance. Microelectron J 88:164–172

    Article  Google Scholar 

  • Li F, Shirahama K, Nisar MA, Köping L, Grzegorzek M (2018) Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors 18(2):679

    Article  Google Scholar 

  • Liciotti D, Bernardini M, Romeo L, Frontoni E (2020) A sequential deep learning application for recognising human activities in smart homes. Neurocomputing 396:501–513

    Article  Google Scholar 

  • Lv T, Wang X, Jin L, Xiao Y, Song M (2020) Margin-based deep learning networks for human activity recognition. Sensors 20(7):1871

    Article  Google Scholar 

  • Mehdiyev N, Lahann J, Emrich A, Enke D, Fettke P, Loos P (2017) Time series classification using deep learning for process planning: a case from the process industry. Proced Comput Sci 114:242–249

    Article  Google Scholar 

  • Micucci D, Mobilio M, Napoletano P (2017) Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Appl Sci 7(10):1101

    Article  Google Scholar 

  • Mozer MC (1998) The neural network house: an environment hat adapts to its inhabitants. In: Proceedings of AAAI Spring Symposium of Intelligent Environments.

  • Mukherjee D, Mondal R, Singh PK, Sarkar R, Bhattacharjee D (2020) EnsemConvNet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications. Multimed Tools Appl 79(41):31663–31690

    Article  Google Scholar 

  • Murad A, Pyun JY (2017) Deep recurrent neural networks for human activity recognition. Sensors 17(11):2556

    Article  Google Scholar 

  • Nweke HF, Teh YW, Al-Garadi MA, Alo UR (2018) Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl 105:233–261

    Article  Google Scholar 

  • Oniga S, Sütő J (2014) Human activity recognition using neural networks. In: Proceedings of the 2014 15th International Carpathian Control Conference (ICCC), IEEE, pp 403–406

  • Osmani V, Balasubramaniam S, Botvich D (2008) Human activity recognition in pervasive health-care: supporting efficient remote collaboration. J Netw Comput Appl 31(4):628–655

    Article  Google Scholar 

  • Paul P, George T (2015) An effective approach for human activity recognition on smartphone. In: 2015 IEEE International Conference on Engineering and Technology (ICETECH), IEEE, pp 1–3

  • Plötz T, Hammerla NY, Olivier P (2011) Feature learning for activity recognition in ubiquitous computing. In: 22nd international joint conference on artificial intelligence, IJCAI 2011, pp 1729–1734

  • Powell HC, Hanson MA, Lach J (2007) A wearable inertial sensing technology for clinical assessment of tremor. In: 2007 IEEE Biomedical Circuits and Systems Conference, IEEE, pp 9–12

  • Ramasamy SR, Roy N (2018) Recent trends in machine learning for human activity recognition—a survey. Wiley Interdiscip Rev 8(4):e1254

    Google Scholar 

  • Ravi D, Wong C, Lo B, Yang GZ (2016) A deep learning approach to on-node sensor data analytics for mobile or wearable devices. IEEE J Biomed Health Inform 21(1):56–64

    Article  Google Scholar 

  • Ravuri S, Stolcke A (2015) Recurrent neural network and LSTM models for lexical utterance classification. In: Sixteenth Annual Conference of the International Speech Communication Association.

  • Ronao CA, Cho SB (2016) Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244

    Article  Google Scholar 

  • Sabatini AM, Martelloni C, Scapellato S, Cavallo F (2005) Assessment of walking features from foot inertial sensing. IEEE Trans Biomed Eng 52(3):486–494

    Article  Google Scholar 

  • Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681

    Article  Google Scholar 

  • Sefen B, Baumbach S, Dengel A, Abdennadher S (2016) Human activity recognition. In: Proceedings of the 8th International Conference on Agents and Artificial Intelligence. SCITEPRESS-Science and Technology Publications, Lda, pp 488–493

  • Shen G, Tan Q, Zhang H, Zeng P, Xu J (2018) Deep learning with gated recurrent unit networks for financial sequence predictions. Proced Comput Sci 131:895–903

    Article  Google Scholar 

  • Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems. Springer, Cham, pp 568–576

    Google Scholar 

  • Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  • Sukor AA, Zakaria A, Rahim NA (2018) Activity recognition using accelerometer sensor and machine learning classifiers. In: 2018 IEEE 14th International Colloquium on Signal Processing and its Applications (CSPA), IEEE, pp 233–238

  • Teng Q, Wang K, Zhang L, He J (2020) The layer-wise training convolutional neural networks using local loss for sensor-based human activity recognition. IEEE Sens J 20(13):7265–7274

    Article  Google Scholar 

  • Torres-Huitzil C, Alvarez-Landero A (2015) Accelerometer-based human activity recognition in smartphones for healthcare services. Mobile health. Springer, Cham, pp 147–169

    Chapter  Google Scholar 

  • Uddin MZ, Hassan MM, Almogren A, Zuair M, Fortino G, Torresen J (2017) A facial expression recognition system using robust face features from depth videos and deep learning. Comput Electr Eng 63:114–125

    Article  Google Scholar 

  • Uddin MZ, Hassan MM, Alsanad A, Savaglio C (2020) A body sensor data fusion and deep recurrent neural network-based behavior recognition approach for robust healthcare. Inform Fus 55:105–115

    Article  Google Scholar 

  • Ullah M, Ullah H, Khan SD, Cheikh FA (2019) Stacked Lstm network for human activity recognition using smartphone data. In: 2019 8th European Workshop on Visual Information Processing (EUVIP), IEEE, pp 175–180

  • Veeriah V, Zhuang N, Qi GJ (2015) Differential recurrent neural networks for action recognition. In: Proceedings of the IEEE international conference on computer vision, pp 4041–4049

  • Vepakomma P, De D, Das SK, Bhansali S (2015) A-Wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities. In: 2015 IEEE 12th International conference on wearable and implantable body sensor networks (BSN), IEEE, pp 1–6

  • Vu TH, Dang A, Dung L, Wang JC (2017) Self-gated recurrent neural networks for human activity recognition on wearable devices. In: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp 179–185

  • Wang J, Chen Y, Hao S, Peng X, Hu L (2019) Deep learning for sensor-based activity recognition: a survey. Pattern Recogn Lett 119:3–11

    Article  Google Scholar 

  • Woodruff RB, Gardial SF (1996) Know your customer: new approaches to customer value and satisfaction blackwell. Cambridge University, Cambridge

    Google Scholar 

  • Woznowski P, King R, Harwin W, Craddock I (2016) A human activity recognition framework for healthcare applications: ontology, labelling strategies, and best practice. In: IoTBD, pp 369–377

  • Wu GE, Xue S (2008) Portable preimpact fall detector with inertial sensors. IEEE Trans Neural Syst Rehabil Eng 16(2):178–183

    Article  Google Scholar 

  • Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866

    Article  Google Scholar 

  • Yager RR (2008) Time series smoothing and OWA aggregation. IEEE Trans Fuzzy Syst 16(4):994–1007

    Article  Google Scholar 

  • Zahin A, Hu RQ (2019) Sensor-based human activity recognition for smart healthcare: a semi-supervised machine learning. International conference on artificial intelligence for communications and networks. Springer, Cham, pp 450–472

    Chapter  Google Scholar 

  • Zainudin MS, Sulaiman MN, Mustapha N, Perumal T (2015) Activity recognition based on accelerometer sensor using combinational classifiers. In: 2015 IEEE Conference on Open Systems (Icos), IEEE, pp 68–73

  • Zeng M, Nguyen LT, Yu B, Mengshoel OJ, Zhu J, Wu P, Zhang J (2014) Convolutional neural networks for human activity recognition using mobile sensors. In: 6th International Conference on Mobile Computing, Applications and Services, IEEE, pp 197–205

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer and Information Technology, Jordan University of Science and Technology, Irbid, Jordan

    Luay Alawneh, Tamam Alsarhan, Mohammad Al-Zinati, Mahmoud Al-Ayyoub & Yaser Jararweh

  2. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Tamam Alsarhan & Hongtao Lu

Authors
  1. Luay Alawneh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tamam Alsarhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammad Al-Zinati
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mahmoud Al-Ayyoub
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yaser Jararweh
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongtao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luay Alawneh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alawneh, L., Alsarhan, T., Al-Zinati, M. et al. Enhancing human activity recognition using deep learning and time series augmented data. J Ambient Intell Human Comput 12, 10565–10580 (2021). https://doi.org/10.1007/s12652-020-02865-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02865-4

Keywords

Search

Navigation