Skip to main content

Advertisement

SpringerLink
Log in

A two-step shapelets based framework for interactional activities recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Human-human interactions recognition has high potential to have a big impact on enabling robots being able to interact with people. Recently, body sensor networks (BSNs) have been widely applied in many fields. In this paper, we investigate the problem of recognizing human interactional activities based on BSNs. Considering that individual actions may contribute differently to the final recognition of interactions, a novel two-step framework that recognizes interactional activities by fusing labels of individual actions is presented. Specifically, shapelets based method is adopted for recognizing individual actions in the first step, and recognition of interactions is realized by weighted fusion of recognized individual actions in the second step. The framework is tested on our newly collected dataset. We mainly compare the performance of the proposed framework with traditional recognition algorithms based framework. Furthermore, feature level fusion is also conducted to verify the effectiveness of the proposed framework. Experimental results show that the proposed framework has achieved promising results with an overall recognition accuracy of 99.44%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Afrasiabi M, Khotanlou H, Gevers T (2020) Spatial-temporal dual-actor cnn for human interaction prediction in video. Multimed Tools Appl 79:20 019–20 038

    Article  Google Scholar 

  2. Alazrai R, Mowafi Y, Lee CG (2015) Anatomical-plane-based representation for human-human interactions analysis. Pattern Recogn 48(8):2346–2363

    Article  Google Scholar 

  3. Attal F, Mohammed S, Dedabrishvili M, Chamroukhi F, Oukhellou L, Amirat Y (2015) Physical human activity recognition using wearable sensors. Sensors 15(12):31 314–31 338

    Article  Google Scholar 

  4. Avci A, Bosch S, Marin-Perianu M, Marin-Perianu R, Havinga P (2011) Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: a survey. In: International conference on architecture of computing systems, pp 167–176

  5. Chen Y, Chao S (2017) Performance analysis of smartphone-sensor behavior for human activity recognition. IEEE Access 5:3095–3110

    Article  Google Scholar 

  6. Chen C, Jafari R, Kehtarnavaz N (2017) A survey of depth and inertial sensor fusion for human action recognition. Multimed Tools Appl 76(3):4405–4425

    Article  Google Scholar 

  7. Chernbumroong S, Cang S, Yu H (2015) Genetic algorithm-based classifiers fusion for multisensor activity recognition of elderly people. IEEE J Biomed Health Inf 19(1):282–289

    Article  Google Scholar 

  8. Coppola C, Faria DR, Nunes U, Bellotto N (2016) Social activity recognition based on probabilistic merging of skeleton features with proximity priors from rgb-d data. In: IEEE/rsj international conference on intelligent robots and systems, pp 5055–5061

  9. Fullerton E, Heller B, Munoz-Organero M (2017) Recognizing human activity in free-living using multiple body-worn accelerometers. IEEE Sens J 17 (16):5290–5297

    Article  Google Scholar 

  10. Gravina R, Alinia P, Ghasemzadeh H, Fortino G (2017) Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges. Inf Fusion 35:68–80

    Article  Google Scholar 

  11. Guo M, Wang Z (2018) Segmentation and recognition of human motion sequences using wearable inertial sensors. Multimed Tools Appl 77:21 201–21 220

    Article  Google Scholar 

  12. Hills J, Lines J, Baranauskas E, Mapp J, Bagnall A (2014) Classification of time series by shapelet transformation. Data Min Knowl Disc 28(4):851–881

    Article  MathSciNet  Google Scholar 

  13. Hsu YL, Yang SC, Chang HC, Lai HC (2018) Human daily and sport activity recognition using a wearable inertial sensor network. IEEE Access 6(99):31 715–31 728

    Article  Google Scholar 

  14. Jain A, Kanhangad V (2018) Human activity classification in smartphones using accelerometer and gyroscope sensors. IEEE Sens J 18(3):1169–1177

    Article  Google Scholar 

  15. Ji X, Wang C, Ju Z (2017) A new framework of human interaction recognition based on multiple stage probability fusion. Appl Sci 7(6):1–16

    Article  Google Scholar 

  16. Keçeli AS, Kaya A, Can AB (2018) Depth features to recognise dyadic interactions. Iet Comput Vis 12(3):331–339

    Article  Google Scholar 

  17. Kong Y, Fu Y (2015) Close human interaction recognition using patch-aware models. IEEE Trans Image Process 25(1):167–178

    Article  MathSciNet  Google Scholar 

  18. Kong Y, Jia Y, Fu Y (2014) Interactive phrases: Semantic descriptions for human interaction recognition. IEEE Trans Pattern Anal Mach Intell 36 (9):1775–1788

    Article  Google Scholar 

  19. Kurban OC, Yıldırım T (2019) Daily motion recognition system by a triaxial accelerometer usable in different positions. IEEE Sens J 19(7):7543–7552

    Article  Google Scholar 

  20. Li M, Leung H (2016) Multiview skeletal interaction recognition using active joint interaction graph. IEEE Trans Multimed 18(11):2293–2302

    Article  Google Scholar 

  21. Li M, Leung H (2019) Multi-view depth-based pairwise feature learning for person-person interaction recognition. Multimed Tools Appl 79:5731–5749

    Article  Google Scholar 

  22. Liu P, Glas DF, Kanda T, Ishiguro H (2016) Data-driven hri: learning social behaviors by example from human-chuman interaction. IEEE Trans Robot 32(4):988–1008

    Article  Google Scholar 

  23. Liu L, Peng Y, Liu M, Huang Z (2015) Sensor-based human activity recognition system with a multilayered model using time series shapelets. Knowl-Based Syst 90:138–152

    Article  Google Scholar 

  24. Liu L, Peng Y, Wang S, Liu M, Huang Z (2016) Complex activity recognition using time series pattern dictionary learned from ubiquitous sensors. Inf Sci 340-341:41–57

    Article  MathSciNet  Google Scholar 

  25. Manzi A, Fiorini L, Limosani R, Dario P, Cavallo F (2018) Two-person activity recognition using skeleton data. Iet Comput Vis 12(1):27–35

    Article  Google Scholar 

  26. Masum AKM, Bahadur EH, Shan-A-Alahi A, Chowdhury MAUZ, Uddin MR, Noman AA (2019) Human activity recognition using accelerometer, gyroscope and magnetometer sensors: Deep neural network approaches. In: 2019 10Th international conference on computing, communication and networking technologies (ICCCNT), pp 1–6

  27. Mohanty M, Sahoo S, Biswal P, Sabut S (2018) Efficient classification of ventricular arrhythmias using feature selection and c4.5 classifier. Biomed Signal Process Control 44:200–208

    Article  Google Scholar 

  28. Peng L, Chen L, Wu X, Guo H, Chen G (2017) Hierarchical complex activity representation and recognition using topic model and classifier level fusion. IEEE Trans Biomed Eng 64(6):1369–1379

    Article  Google Scholar 

  29. Portugal D, Santos L, Alvito P, Dias J, Samaras G, Christodoulou E (2015) Socialrobot: an interactive mobile robot for elderly home care. In: IEEE/SICE International symposium on system integration

  30. Qiu S, Wang Z, Zhao H, Liu L, Li J, Jiang Y, Fortino G (2018) Body sensor network based robust gait analysis: Toward clinical and at home use. IEEE Sens J PP(99):1–1

    Google Scholar 

  31. Rashidi P, Mihailidis A (2013) A survey on ambient-assisted living tools for older adults. IEEE J Biomed Health Informatics 17(3):579–590

    Article  Google Scholar 

  32. Robertas D, Vasiljevas M, Šalkevičius J, Woźniak M (2016) Human activity recognition in aal environments using random projections. Comput Math Methods Med 2016:1–17

    MathSciNet  MATH  Google Scholar 

  33. Wang H, Li C, Sun H, Guo Z, Bai Y (2018) Shapelet classification algorithm based on efficient subsequence matching. Data Sci J 17(5):1–12

    Google Scholar 

  34. Wang Z, Wu DH, Chen JM, Ghoneim A, Hossain MA (2016) A triaxial accelerometer-based human activity recognition via eemd-based features and game-theory-based feature selection. IEEE Sens J 16(9):3198–3207

    Article  Google Scholar 

  35. Wei Y, Jiao L, Wang S, Chen Y, Liu D (2015) Time series classification with max-correlation and min-redundancy shapelets transformation. In: International conference on identification, information, and knowledge in the internet of things, pp 7–12

  36. Wu H, Shao J, Xu X, Ji Y, Shen F, Shen HT (2018) Recognition and detection of two-person interactive actions using automatically selected skeleton features. IEEE Trans Human-Mach Syst 48(3):304–310

    Article  Google Scholar 

  37. Yang Y, Deng Q, Shen F, Zhao J, Luo C (2017) A shapelet learning method for time series classification. In: IEEE International conference on TOOLS with artificial intelligence, CA, pp 423–430

  38. Yatbaz HY, Eraslan S, Yesilada Y, Ever E (2019) Activity recognition using binary sensors for elderly people living alone: scanpath trend analysis approach. IEEE Sens J 19(17):7575–7582

    Article  Google Scholar 

  39. Ye L, Keogh E (2009) Time series shapelets: a new primitive for data mining. In: ACM SIGKDD International conference on knowledge discovery and data mining, Paris, pp 947–956

  40. Ye L, Keogh E (2011) Time series shapelets: a novel technique that allows accurate, interpretable and fast classification. Data Min Knowl Disc 22 (1-2):149–182

    Article  MathSciNet  Google Scholar 

  41. Ye L, Nie L, Li L, Rosenblum DS (2016) From action to activity: Sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  42. Yuan J-D, Wang Z-H, Han M (2014) A discriminative shapelets transformation for time series classification. Int J Pattern Recognit Artif Intell 28 (06):307–318

    Article  Google Scholar 

  43. Zalewski W, Silva F, Maletzke AG, Ferrero CA (2016) Exploring shapelet transformation for time series classification in decision trees. Knowl-Based Syst 112:80–91

    Article  Google Scholar 

  44. Zhang SC, Li XL, Zong M, Zhu XF, Wang RL (2017) Efficient knn classification with different numbers of nearest neighbors. IEEE Trans Neural Netw Learn Syst PP(99):1–12

    Article  Google Scholar 

  45. Zhao H, Wang Z, Qiu S, Wang J, Xu F, Wang Z, Shen Y (2019) Adaptive gait detection based on foot-mounted inertial sensors and multi-sensor fusion. Inf Fusion 52:157–166

    Article  Google Scholar 

  46. Zlatintsi A, Rodomagoulakis I, Pitsikalis V, Koutras P, Kardaris N, Papageorgiou X, Tzafestas C, Maragos P (2017) Social human-robot interaction for the elderly: Two real-life use cases. In: Companion of the acm/ieee international conference on human-robot interaction

Download references

Acknowledgments

This work was supported in part by National Natural Science Foundation of China under Grant 61873044, Grant 61903062, and Grant 61803072, in part by Natural Science Foundation of Liaoning Province under Grant No. 2019-MS-056, in part by Dalian Science and Technology Innovation fund 2018J12SN077, 2019J13SN99 and 2020JJ27SN067, and in part by Fundamental Research Funds for the Central Universities under Grant DUT21YG125.

Author information

Authors and Affiliations

  1. School of Control Science and Engineering, Dalian University of Technology, 116024, Dalian, China

    Ning Yang, Zhelong Wang, Hongyu Zhao, Xin Shi & Sen Qiu

Authors
  1. Ning Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhelong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongyu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sen Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ning Yang.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, N., Wang, Z., Zhao, H. et al. A two-step shapelets based framework for interactional activities recognition. Multimed Tools Appl 81, 17595–17614 (2022). https://doi.org/10.1007/s11042-022-11987-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-11987-0

Keywords

Search

Navigation