Skip to main content
SpringerLink
Log in

Primitive activity recognition from short sequences of sensory data

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Activity recognition from mobile device sensors and wearables is attracting more attention from the research community due to the widespread adoption of these devices and the unique opportunity they provide for understanding user’s behavior leading to novel services and improvements in the delivery of existing ones. Approaches to tackle this problem either rely on predefined statistical features of sensor data streams or feature learning with the latter providing higher accuracies in most cases. Deep learning methods proved more effective than traditional approaches to feature learning in multiple studies. This paper presents a novel end-to-end trainable deep architecture that utilizes multiple convolutional neural networks (CNN), late fusion and extensive layer bypassing. The proposed method can easily accommodate multiple sensors and signal representations. The proposed approach is validated on eight publicly available datasets using a variety of evaluation conditions showing that it outperforms state-of-the-art methods in six of them.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. The authors would like to thank M. Alsheikh for providing us with the source code of their method enabling the comparative experiment reported here

References

  1. Abdallah ZS, Gaber MM, Srinivasan B, Krishnaswamy S (2015) Adaptive mobile activity recognition system with evolving data streams. Neurocomputing 150(Part A):304–317

    Article  Google Scholar 

  2. Alsheikh MA, Selim A, Niyato D, Doyle L, Lin S, Tan HP (2016) Deep activity recognition models with triaxial accelerometers. In: The workshops of the thirtieth AAAI conference on artificial intelligence, pp 1–8

  3. Altun K, Barshan B (2010) Human activity recognition using inertial/magnetic sensor units. In: International workshop on human behavior understanding. Springer, pp 38–51

  4. Anderez DO, Appiah K, Lotfi A, Langesiepen C (2017) A hierarchical approach towards activity recognition. In: Proceedings of the 10th international conference on PErvasive technologies related to assistive environments. ACM, pp 269–274

  5. Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2013) A public domain dataset for human activity recognition using smartphones. In: European symposium on artificial neural networks, computational intelligence and machine learning, pp 24–26

  6. Anwar S, Hwang K, Sung W (2017) Structured pruning of deep convolutional neural networks. ACM J Emerg Technol Comput Syst (JETC) 13(3):32

    Google Scholar 

  7. Barshan B, Yüksek MC (2014) Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. Comput J 57(11):1649–1667

    Article  Google Scholar 

  8. Borazio M, Van Laerhoven K (2013) Using time use with mobile sensor data: a road to practical mobile activity recognition?. In: Proceedings of the 12th international conference on mobile and ubiquitous multimedia, MUM ’13. ACM, New York, pp 20:1–20:10

  9. Catal C, Tufekci S, Pirmit E, Kocabag G (2015) On the use of ensemble of classifiers for accelerometer-based activity recognition. Applied Soft Computing Journal 37:1018–1022

    Article  Google Scholar 

  10. Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C (Appl Rev) 42(6):790–808

    Article  Google Scholar 

  11. Cornwell BR, Carver FW, Coppola R, Johnson L, Alvarez R, Grillon C (2008) Evoked amygdala responses to negative faces revealed by adaptive meg beamformers. Brain Res 1244:103–112

    Article  Google Scholar 

  12. Dharia S, Jain V, Patel J, Vora J, Chawla S, Eirinaki M (2016) PRO-Fit: a personalized fitness assistant framework. Int J Softw Eng Knowl Eng 26(9):386–389

    Article  Google Scholar 

  13. Forster K, Roggen D, Troster G (2009) Unsupervised classifier self-calibration through repeated context occurences: is there robustness against sensor displacement to gain?. In: 2009 international symposium on wearable computers, pp 77–84

  14. Garvert MM, Friston KJ, Dolan RJ, Garrido MI (2014) Subcortical amygdala pathways enable rapid face processing. Neuroimage 102:309–316

    Article  Google Scholar 

  15. Guan Y, Plötz T (2017) Ensembles of deep lstm learners for activity recognition using wearables. Proc ACM Interact Mob Wearable Ubiquitous Technol 1(2):11:1–11:28

    Article  Google Scholar 

  16. Hammerla NY, Halloran S, Plötz T (2016) Deep, convolutional, and recurrent models for human activity recognition using wearables. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, IJCAI’16, pp 1533–1540

  17. Hasan M, Roy-Chowdhury AK (2015) A continuous learning framework for activity recognition using deep hybrid feature models. IEEE Trans Multimedia 17(11):1909–1922

    Article  Google Scholar 

  18. Karantonis DM, Narayanan MR, Mathie M, Lovell NH, Celler BG (2006) Implementation of a real-time human movement classifier using a triaxial accelerometer for ambulatory monitoring. IEEE Trans Inf Technol Biomed 10(1):156–167

    Article  Google Scholar 

  19. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  20. Kreil M, Sick B, Lukowicz P (2016) Coping with variability in motion based activity recognition. In: Proceedings of the 3rd international workshop on sensor-based activity recognition and interaction, iWOAR ’16. ACM, New York, pp 4:1–4:8

  21. Kwapisz JR, Weiss GM, Moore SA (2011) Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter 12(2):74–82

    Article  Google Scholar 

  22. Liu S, Deng W (2015) Very deep convolutional neural network based image classification using small training sample size. In: 2015 3rd IAPR Asian conference on pattern recognition (ACPR), pp 730–734

  23. Lockhart JW, Pulickal T, Weiss GM (2012) Applications of mobile activity recognition. In: Proceedings of the 2012 ACM conference on ubiquitous computing. ACM, pp 1054–1058

  24. Lu DN, Nguyen TT, Ngo TTT, Nguyen TH, Nguyen HN (2017) Mobile online activity recognition system based on smartphone sensors. Springer International Publishing, Cham, pp 357–366

    Google Scholar 

  25. Mizell D (2003) Using gravity to estimate accelerometer orientation. In: Seventh IEEE international symposium on wearable computers, 2003. Proceedings. IEEE, pp 252–253

  26. Ordóñez FJ, Roggen D (2016) Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1):115

    Article  Google Scholar 

  27. Plötz T, Hammerla NY, Olivier P (2011) Feature learning for activity recognition in ubiquitous computing. In: Proceeding IJCAI’11 proceedings of the twenty-second international joint conference on artificial intelligence, vol 2, pp 1729–1734

  28. Raman N, Maybank SJ (2016) Activity recognition using a supervised non-parametric hierarchical hmm. Neurocomputing 199:163–177

    Article  Google Scholar 

  29. Roggen D, Calatroni A, Rossi M, Holleczek T, Forster K, Troster G, Lukowicz P, Bannach D, Pirkl G, Ferscha A, Doppler J, Holzmann C, Kurz M, Holl G, Chavarriaga R, Sagha H, Bayati H, Creatura M, Millan JDR (2010) Collecting complex activity datasets in highly rich networked sensor environments. In: 2010 Seventh international conference on networked sensing systems (INSS), July 2010. IEEE, pp 233–240

  30. Ronao CA, Cho SB (2016) Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244

    Article  Google Scholar 

  31. Sagha H, Digumarti ST, Millán JDR, Chavarriaga R, Calatroni A, Roggen D, Tröster G (2011) Benchmarking classification techniques using the opportunity human activity dataset. In: 2011 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 36–40

  32. San-Segundo R, Montero JM, Barra-Chicote R, Fernández F, Pardo JM (2016) Feature extraction from smartphone inertial signals for human activity segmentation. Signal Process 120:359–372

    Article  Google Scholar 

  33. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems, pp 568–576

  34. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (ICLR)

  35. Urban G, Geras KJ, Kahou SE, Aslan O, Wang S, Caruana R, Mohamed A, Philipose M, Richardson M (2016) Do deep convolutional nets really need to be deep and convolutional? arXiv:1603.05691

  36. Veenendaal A, Daly E, Jones E, Gang Z, Vartak S, Patwardhan RS (2016) Sensor tracked points and HMM based classifier for human action recognition. Computer Science and Emerging Research Journal 5:4–8

    Google Scholar 

  37. Weiss GM, Lockhart JW (2011) Identifying user traits by mining smart phone accelerometer data. In: Proceedings of the fifth international workshop on knowledge discovery from sensor data, SensorKDD2011, pp 61–69

  38. Weiss GM, Lockhart JW (2012) The impact of personalization on smartphone-based activity recognition. In: AAAI workshop on activity context representation: techniques and languages, pp 98–104

  39. Weiss GM, Lockhart JW, Pulickal TT, McHugh PT, Ronan IH, Timko JL (2016) Actitracker: a smartphone-based activity recognition system for improving health and well-being. In: 2016 IEEE international conference on data science and advanced analytics (DSAA). IEEE, pp 682–688

  40. Yu J, Lukefahr A, Palframan D, Dasika G, Das R, Mahlke S (2017) Scalpel: customizing dnn pruning to the underlying hardware parallelism. In: Proceedings of the 44th annual international symposium on computer architecture. ACM, pp 548–560

  41. Zagoruyko S, Lerer A, Lin TY, Pinheiro PO, Gross S, Chintala S, Dollár P (2016) A multipath network for object detection. In: British machine vision conference. New York, pp 15.1–15.12

  42. Zappi P, Lombriser C, Stiefmeier T, Farella E, Roggen D, Benini L, Troster G (2008) Activity recognition from on-body sensors: accuracy-power trade-off by dynamic sensor selection. Lect Notes Comput Sci 4913:17

    Article  Google Scholar 

  43. Zappi P, Stiefmeier T, Farella E, Roggen D, Benini L, Troster G (2007) Activity recognition from on-body sensors by classifier fusion: sensor scalability and robustness. In: 2007 3rd international conference on intelligent sensors, sensor networks and information, pp 281–286

  44. Zeng M, Nguyen LT, Yu B, Mengshoel OJ, Zhu J, Wu P, Zhang J (2014) Convolutional neural networks for human activity recognition using mobile sensors. In: Proceedings of the 6th international conference on mobile computing, applications and services, ICST, vol 6, pp 197–205

  45. Zhu Z, Blanke U, Calatroni A, Tröster G (2013) Human activity recognition using social media data. In: Proceedings of the 12th international conference on mobile and ubiquitous multimedia, MUM ’13. ACM, New York, pp 21:1–21:10

Download references

Author information

Authors and Affiliations

  1. Assiut University, Asyut, Egypt

    Yasser Mohammad

  2. KDDI Research Inc., Fujimino, Japan

    Yasser Mohammad, Kazunori Matsumoto & Keiichiro Hoashi

Authors
  1. Yasser Mohammad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kazunori Matsumoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Keiichiro Hoashi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yasser Mohammad.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohammad, Y., Matsumoto, K. & Hoashi, K. Primitive activity recognition from short sequences of sensory data. Appl Intell 48, 3748–3761 (2018). https://doi.org/10.1007/s10489-018-1166-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-018-1166-6

Keywords

Search

Navigation