Skip to main content
SpringerLink
Log in

Sensor-data augmentation for human activity recognition with time-warping and data masking

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Human activity recognition (HAR) using an accelerometer can provide valuable information for understanding user context. Therefore, several studies have been conducted using deep learning to increase the recognition rate of activity classification. However, the existing dataset that is publicly available for HAR tasks contains limited data. Previous works have applied data augmentation methods that simply transform the entire accelerometer-signal dataset. However, the label of the augmented signal cannot be easily recognized by humans, and the augmentation methods cannot ensure that the label of the signal is preserved. Therefore, we propose a novel data augmentation method that reflects the characteristics of the sensor signal and can preserve the label of the augmented signal by generating partially occluded data of the accelerometer signals. To generate the augmented data, we apply time-warping, which deforms the time-series data in the time direction. We handle jittering effects and subsequently apply data masking to drop out a part of the input signals. We compare the performance of the proposed augmentation method with that of conventional methods by using two public datasets and an activity recognition model based on convolutional neural networks. The experimental results show that the proposed augmentation method improves the recognition rate of the activity classification model, regardless of the dataset. Additionally, the proposed method shows superior performance over conventional methods on the two datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Abu Alsheikh M, Selim A, Niyato D, Doyle L, Lin S, Tan H (2016) Deep activity recognition models with triaxial accelerometers. In: AAAI Conference on artificial intelligence, AI Access Foundation, United States, vol WS-16-01 - WS-16-15. pp 8–13

  2. Avilés-Cruz C, Ferreyra-Ramírez A, Zúñiga-López A, Villegas-Cortéz J (2019) Coarse-fine convolutional deep-learning strategy for human activity recognition. Sensors 19(7). https://doi.org/10.3390/s19071556, https://www.mdpi.com/1424-8220/19/7/1556

  3. Chollet F, et al. (2015) Keras: Deep learning library for theano and tensorflow. https://keras io/k 7(8):T1

  4. Chung S, Lim J, Noh KJ, Kim G, Jeong H (2019) Sensor data acquisition and multimodal sensor fusion for human activity recognition using deep learning. Sensors 19(7). https://doi.org/10.3390/s19071716. https://www.mdpi.com/1424-8220/19/7/1716

  5. Cui X, Goel V, Kingsbury B (2015) Data augmentation for deep neural network acoustic modeling. IEEE/ACM Trans Audio Speech Lang Process (TASLP) 23(9):1469–1477

    Article  Google Scholar 

  6. DeVries T, Taylor GW (2017) Improved regularization of convolutional neural networks with cutout. arXiv:1708.04552

  7. Dwibedi D, Misra I, Hebert M (2017) Cut, paste and learn: Surprisingly easy synthesis for instance detection. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp 1310–1319

  8. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580

  9. Inoue M, Inoue S, Nishida T (2018) Deep recurrent neural network for mobile human activity recognition with high throughput. Artif Life Robot 23 (2):173–185. https://doi.org/10.1007/s10015-017-0422-x

    Article  Google Scholar 

  10. Jeong CY, Kim M (2019) An energy-efficient method for human activity recognition with segment-level change detection and deep learning. Sensors 19(17). https://doi.org/10.3390/s19173688.https://www.mdpi.com/1424-8220/19/17/3688

  11. Jiang W, Yin Z (2015) Human activity recognition using wearable sensors by deep convolutional neural networks. In: Proceedings of the 23rd ACM International conference on multimedia, ACM, New York, NY, USA, MM ’15. https://doi.org/10.1145/2733373.2806333, pp 1307–1310

  12. Kalouris G, Zacharaki EI, Megalooikonomou V (2019) Improving cnn-based activity recognition by data augmentation and transfer learning. In: 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), vol 1, pp 1387–1394

  13. Kim M, Jeong CY (2020) Label-preserving data augmentation for mobile sensor data. Multid Syst Signal Process. https://doi.org/10.1007/s11045-020-00731-2

  14. Kim M, Jeong CY, Shin HC (2018) Activity recognition using fully convolutional network from smartphone accelerometer. In: 2018 International conference on information and communication technology convergence (ICTC). https://doi.org/10.1109/ICTC.2018.8539419, pp 1482–1484

  15. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. arXiv:1412.6980

  16. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  17. Kwapisz JR, Weiss GM, Moore SA (2011) Activity recognition using cell phone accelerometers. SIGKDD Explor Newsl 12(2):74–82. https://doi.org/10.1145/1964897.1964918

    Article  Google Scholar 

  18. Mathur A, Zhang T, Bhattacharya S, Veličković P, Joffe L, Lane ND, Kawsar F, Lió P (2018) Using deep data augmentation training to address software and hardware heterogeneities in wearable and smartphone sensing devices. In: Proceedings of the 17th ACM/IEEE International conference on information processing in sensor networks, IEEE Press, IPSN ’18. https://doi.org/10.1109/IPSN.2018.00048, pp 200–211

  19. Micucci D, Mobilio M, Napoletano P (2017) Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Applied Sciences 7(10). https://doi.org/10.3390/app7101101, http://www.mdpi.com/2076-3417/7/10/1101

  20. Moya Rueda F, Grzeszick R, Fink GA, Feldhorst S, ten Hompel M (2018) Convolutional neural networks for human activity recognition using body-worn sensors. Informatics 5(2). https://doi.org/10.3390/informatics5020026. http://www.mdpi.com/2227-9709/5/2/26

  21. Mukherjee D, Mondal R, Singh PK, Sarkar R, Bhattacharjee D (2020) Ensemconvnet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications. Multimed Tools Appl https://doi.org/10.1007/s11042-020-09537-7

  22. Murad A, Pyun JY (2017) Deep recurrent neural networks for human activity recognition. Sensors 17(11). https://doi.org/10.3390/s17112556. http://www.mdpi.com/1424-8220/17/11/2556

  23. Nanni L, Maguolo G, Paci M (2019) Data augmentation approaches for improving animal audio classification. arXiv:191207756

  24. Ohashi H, Al-Naser MOA, Ahmed S, Akiyama T, Sato T, Nguyen P, Nakamura K, Dengel A (2017) Augmenting wearable sensor data with physical constraint for dnn-based human-action recognition. In: Time series workshop. Time Series Workshop @ ICML, befindet sich ICML 2017 August 11-11, Sydney, Australia

  25. Pan Y, He F, Yu H (2020) Learning social representations with deep autoencoder for recommender system. World Wide Web 23(4):2259–2279. https://doi.org/10.1007/s11280-020-00793-z

    Article  Google Scholar 

  26. Park DS, Chan W, Zhang Y, Chiu CC, Zoph B, Cubuk ED, Le QV (2019) Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech 2019. https://doi.org/10.21437/interspeech.2019-2680

  27. Piczak KJ (2015) Environmental sound classification with convolutional neural networks. In: 2015 IEEE 25th International workshop on machine learning for signal processing (MLSP). IEEE, pp 1–6

  28. Quan Q (2020) He F. A multi-phase blending method with incremental intensity for training detection networks. Vis Comput, Li H. https://doi.org/10.1007/s00371-020-01796-7

    Google Scholar 

  29. Ravì D, Wong C, Lo B, Yang G (2017) A deep learning approach to on-node sensor data analytics for mobile or wearable devices. IEEE J Biomed Health Inf 21(1):56–64. https://doi.org/10.1109/JBHI.2016.2633287

    Article  Google Scholar 

  30. Salamon J, Bello JP (2017) Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Process Lett 24(3):279–283

    Article  Google Scholar 

  31. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  32. Stisen A, Blunck H, Bhattacharya S, Prentow TS, Kjærgaard MB, Dey A, Sonne T, Jensen MM (2015) Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In: Proceedings of the 13th ACM Conference on embedded networked sensor systems, association for computing machinery, New York, NY, USA, SenSys ’15. https://doi.org/10.1145/2809695.2809718, pp 127–140

  33. Um TT, Pfister FMJ, Pichler D, Endo S, Lang M, Hirche S, Fietzek U, Kulić D (2017) Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, Association for Computing Machinery, New York, NY, USA, ICMI ’17. https://doi.org/10.1145/3136755.3136817, pp 216–220

  34. Weiss G, Lockhart J (2012) The impact of personalization on smartphone-based activity recognition. In: Twenty-sixth AAAI conference on artificial intelligence. https://www.aaai.org/ocs/index.php/WS/AAAIW12/paper/view/5203/5564

  35. Yoo SB, Han M (2020) Temporal matching prior network for vehicle license plate detection and recognition in videos. ETRI J 42(3):411–419. https://doi.org/10.4218/etrij.2019-0245

    Article  Google Scholar 

  36. Yun K, Kwon Y, Oh S, Moon J, Park J (2019) Vision-based garbage dumping action detection for real-world surveillance platform. ETRI J 41 (4):494–505. https://doi.org/10.4218/etrij.2018-0520

    Article  Google Scholar 

  37. Zeng M, Nguyen LT, Yu B, Mengshoel OJ, Zhu J, Wu P, Zhang J (2014) Convolutional neural networks for human activity recognition using mobile sensors. In: 6th International conference on mobile computing applications and services. https://doi.org/10.4108/icst.mobicase.2014.257786, pp 197–205

  38. Zhang S, He F (2020) Drcdn: learning deep residual convolutional dehazing networks. Vis Comput 36 (9):1797–1808. https://doi.org/10.1007/s00371-019-01774-8

    Article  Google Scholar 

  39. Zhang S, He F, Ren W (2020) Nldn: Non-local dehazing network for dense haze removal. Neurocomputing 410:363–373. https://doi.org/10.1016/j.neucom.2020.06.041. http://www.sciencedirect.com/science/article/pii/S0925231220310122

    Article  Google Scholar 

Download references

Acknowledgements

This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [1711103127, Development of Human Enhancement Technology for auditory and muscle support] and Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government (MSIT) [20ZS1100, Core Technology Research for Self-Improving Integrated Artificial Intelligence System].

Author information

Authors and Affiliations

  1. Human Enhancement and Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics and Telecommunications Research Institute (ETRI), Daejeon, Korea

    Chi Yoon Jeong, Hyung Cheol Shin & Mooseop Kim

Authors
  1. Chi Yoon Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyung Cheol Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mooseop Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mooseop Kim.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jeong, C.Y., Shin, H.C. & Kim, M. Sensor-data augmentation for human activity recognition with time-warping and data masking. Multimed Tools Appl 80, 20991–21009 (2021). https://doi.org/10.1007/s11042-021-10600-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10600-0

Keywords

Search

Navigation