Skip to main content
SpringerLink
Log in

Automatic human posture estimation for sport activity recognition with robust body parts detection and entropy markov model

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automated human posture estimation (A-HPE) systems need delicate methods for detecting body parts and selecting cues based on marker-less sensors to effectively recognize complex activity motions. Recognition of human activities using vision sensors is a challenging issue due to variations in illumination conditions and complex movements during the monitoring of sports and fitness exercises. In this paper, we propose a novel A-HPE method that intelligently identifies human behaviours by utilizing saliency silhouette detection, robust body parts model and multidimensional cues from full-body silhouettes followed by an entropy Markov model. Initially, images are pre-processed and noise is removed to obtain a robust silhouette. Body parts models are then used to extract twelve key body parts. These key body parts are further optimized to assist the generation of multidimensional cues. These cues include energy, optical flow and distinctive values that are fed into quadratic discriminant analysis to discriminate cues which help in the recognition of actions. Finally, these optimized patterns are further processed by a maximum entropy Markov model as a recognizer engine based on transition and emission probability values for activity recognition. For evaluation, we used a leave-one-out cross validation scheme and the results outperformed existing well-known statistical state-of-the-art methods by achieving better body parts detection and higher recognition accuracy over four benchmark datasets. The proposed method will be useful for man-machine interactions such as 3D interactive games, virtual reality, service robots, e-health fitness, and security surveillance.

Design model of automatic posture estimation and action recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

References

  1. Ahmed A, Jalal A, Kim K (2020) A novel statistical method for scene classification based on multi–object categorization and logistic regression. Sensors

  2. Al-Ghannam R, Al–Dossari H (2016) Prayer Activity Monitoring and Recognition Using Acceleration Features with Mobile Phone. Arabian J Sci Eng 41:4967–4979

  3. Andriluka M, Pishchulin L, Gehler P, Schiele (2014) 2D Human Pose Estimation: New Benchmark and State of the Art Analysis. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2014.471

  4. Badar S, Jalal A, Kim K (2020) Wearable Inertial Sensors for Daily Activity Analysis Based on Adam Optimization and the Maximum Entropy Markov Model. Entropy 22(5):1–19

  5. Bay H, Tuytelaars T, Gool LV (2006) SURF: Speeded up robust features.European Conference of Computer Vision. https://doi.org/10.1007/11744023

  6. Beigi H (2010) Voice: technologies and algorithms for biometrics applications Homayoon Beigi. IEEE Courses: Bioengineering

  7. Chen H, Mcgurr M (2014) Improved color and intensity patch segmentation for human full–body and body–parts detection and tracking. IEEE: Proceedings of AVSS. https://doi.org/10.1109/AVSS.2014.6918695

  8. DAS S, Chuadhary A, Bremond F, Thonnat M (2019) Where to focus on for human action recognition?. IEEE Winter Conference on Applications of computer vision. https://doi.org/10.1109/WACV.2019.00015

  9. Daniel W, Remi R, Edmond B (2006) Free Viewpoint Action Recognition using Motion History Volumes. Comput Vis Image Underst (CVIU) 104:249–257

  10. Dargazany A, Nicolescu M (2012) Human body parts tracking using torso tracking: applications to activity recognition. Proceedings of ITNG. arXiv:1907.05281

  11. Dawn DD, Shaikh SH (2016) A comprehensive survey of human action recognition with spatio–temporal interest point (STIP) detector. The Vis Comput 32:289–306

  12. Fei M, Ju Z, Zhen X, Li J (2017) Real-time Visual Tracking based on Improved Perceptual Hashing [J]. Multimed Tools Appl 76(3):4617–4634

  13. Guo Y, Yue X, Yan G (2013) Salient region detection based on multi–resolution. IEEE: International Conference on Machine learning and Cybernetics. https://doi.org/10.1109/ICMLC.2013.6890422

  14. Han Y, Chung S, Ambikapathi A, Chan J, Lin W, Su S (2018) Robust human action recognition using global spatial–temporal attention for human skeleton data. Proceedings of IJCNN. DOI: https://doi.org/10.1109/IJCNN.2018.8489386

  15. Hu Z, Lin X, Yan H (2006) Torso Detection in Static Images. IEEE: International Conference on Signal. https://doi.org/10.1109/ICOSP.2006.345837. Processing

  16. Hussain I (2019) AAMAZ Human Action Recognition Dataset, Kaggle

  17. Ignatov A (2018) Real–time human activity recognition from accelerometer data using Convolutional Neural Networks. Appl Soft Comput 62:915–922

  18. Jalal A, Zia-Uddin M, Kim T (2012) Depth Video–based Human Activity Recognition System Using Translation and Scaling Invariant Features for Life Logging at Smart Home, IEEE Transaction on Consumer Electronics, ISSN: 0098–3063 58(3):863–871

  19. Jalal A, Sharif N, Kim J et al (2013) Human activity recognition via recognized body parts of human depth silhouettes for residents monitoring services at smart homes. Indoor Built Environ 22:271–279

  20. Jalal A, Kamal S, Kim D (2014) A depth video sensor–based life–logging human activity recognition system for elderly care in smart indoor environments. Sensors 14(7):11735–11759

  21. Jalal A, Kim Y, Kim D (2014) Ridge body parts features for human pose estimation and recognition from RGB–D video data. Proceedings of the IEEE International Conference on computing, communication and networking technologies, pp. 1–6

  22. Jalal A, Mahmood M, Sidduqi M (2018) Robust spatio–temporal features for human interaction recognition via artificial neural network, IEEE conference on International Conference on Frontiers of information technology

  23. Jalal A, Nadeem A, Bobasu S (2019) Human body parts estimation and detection for physical sports movements. IEEE International Conference on Communication, Computing and Digital Systems

  24. Jalal A, Khalid N, Kim K (2020) Automatic Recognition of Human Interaction via Hybrid Descriptors and Maximum Entropy Markov Model Using Depth Sensors. Entropy

  25. Jaouedi N, Boujnah N, Bouhlel MS (2019) A new hybrid deep learning model for human action recognition. Journal of King Saud University – Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2019.09.004

  26. Johnson E (2010) Clustered Pose and Non–linear Appearance Models for Human Pose Estimation. Proceedings of the British Machine Vision Conference. https://doi.org/10.5244/C.24.12

  27. Kim Y, Kim D Real-time dance evaluation by markerless human pose estimation. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-6068-4

  28. Khan MUS, Abbas A, Ali M (2018) On the Correlation of Sensor Location and Human Activity Recognition in Body Area Networks (BANs). IEEE Syst J 12:82–91

  29. Li J, Li X, Tao D (2008) KPCA for Semantic Object Extraction in Images [J] Pattern Recognition 41(10):3244–3250

  30. Li N, Wen L, Dong X (2015) Visual recognition by learning from web data: A weakly supervised domain generalization approach. IEEE Conf. Comput. Vis. Pattern Recognit (CVPR) https://doi.org/10.1109/CVPR.2015.7298894

  31. Li G, Tang H, Sun Y et al (2019) Hand gesture recognition based on convolution neural network. Cluster Comput 22(Supplement 2): 2719–2729. https://doi.org/10.1007/s10586-017-1435-x

  32. Li C, Zhang B, Chen C et al (2019) Deep Manifold Structure Transfer for Action Recognition. IEEE transactions on image processing 28(9) https://doi.org/10.1109/TIP.2019.2912357

  33. Liu M, Liu H, Chen C (2017) Enhanced skeleton visualization for view invariant human action recognition. Pattern Recogn 68:346–362

  34. Liu C, Yuen PC (2011) A Boosted Co–Training Algorithm for Human Action Recognition. IEEE Trans Circ Syst Video Technol 21:1203–1213

  35. Liu M, Liu H, Sun Q, Zhang T, Ding R (2016) Salient pairwise spatio–temporal interest points for real–time activity recognition. CAAI Trans Intell Technol 1:14–29

  36. Liu T, Stathaki T (2016) Fast head–shoulder proposal for deformable part model based pedestrian detection. IEEE International Conference on Digital Signal Processing (DSP). https://doi.org/10.1109/ICDSP.2016.7868599

  37. Liu F, Xu X, Qiu S, Qing C, Tao D (2016) Simple to complex transfer learning for action recognition. IEEE Trans Image Process 25:949–960

  38. Luvizon DC, Hedi T, David P (2017) Learning features combination for human action recognition from skeleton sequences. Pattern Recogn Lett 99:13–20

  39. Madabhushi A, Aggarwal J (1999) A bayesian approach to human activity recognition, IEEE Visual Surveillance. https://doi.org/10.1109/VS.1999.780265

  40. Mahmood M, Jalal A, Kim K (2020) WHITE STAG Model: Wise Human Interaction Tracking and Estimation (WHITE) using Spatio–temporal and Angular–geometric (STAG) Descriptors, Multimedia Tools and Applications

  41. Manzi A, Moschetti A, Limosani R, Fiorini L, Cavallo F (2018) Enhancing Activity Recognition of Self–Localized Robot Through Depth Camera and Wearable Sensors. IEEE Sens J 18:9324–9331

  42. Milanova M, Ali S, Al-Rizzo H, Fox VL (2015) Human action Recognition: Contour–based and silhouette–based Approaches. Springer Cham. https://doi.org/10.1007/978-3-319-11430-9

  43. Mojarrad M, Dezfouli M, Rahmani A (2008) Feature’s Extraction of Human Body Composition in Images by Segmentation Method. Pwaset 35:267–270

  44. Nadeem A, Jalal A, Kim K (2020) Human actions tracking and recognition based on body parts detection via Artificial neural network. IEEE International Conference on Advancements in computational sciences

  45. Nguyen ND, Bui DT, Truong PH, Jeong GM (2018) Classification of Five Ambulatory Activities Regarding Stair and Incline Walking Using Smart Shoes. IEEE Sensors Journal. https://doi.org/10.1109/JSEN.2018.2837674

  46. Osterland S, Weber J (2019) Analytical analysis of single–stage pressure relief valves. Int J Hydromechatron 2(1):32–53

  47. Quaid M, Jalal A (2019) Wearable Sensors based Human Behavioral Pattern Recognition using Statistical Features and Reweighted Genetic Algorithm. Multimedia Tools and Applications

  48. Rezaie H, Ghassemian M (2017) An Adaptive Algorithm to Improve Energy Efficiency in Wearable Activity Recognition Systems. IEEE Sens J 17:5315–5323

  49. Riemenschneider H, Donoser M, Bischof H (2009) Bag of Optical Flow Volumes for Image Sequence Recognition. British Machine Vision Conference. https://doi.org/10.5244/C.23.28

  50. Sadanand S, Corso JJ (2012) Action bank: A high–level representation of activity invideo. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit, https://doi.org/10.1109/CVPR.2012.6247806

  51. San-Segundo R, Blunck H, Moreno-Pimentel J, Stisen A, Gil-Martín M (2018) Robust Human Activity Recognition using smartwatches and smartphones. Eng Appl Artif Intell 72:190–202

  52. Shebiah RN, Sangari AA (2019) Classification of human body parts using histogram of oriented gradients. Proceedings of ICACCS. https://doi.org/10.1109/ICACCS.2019.8728328

  53. Shokri M, Tavakoli K (2019) A review on the artificial neural network approach to analysis and prediction of seismic damage in infrastructure. Int J Hydromechatron 2(4):178–196

  54. Sun Y et al (2020) Intelligent human computer interaction based on non redundant EMG signal. Alexandria Engineering Journal https://doi.org/10.1016/j.aej.2020.01.015

  55. Susan S, Agrawal P, Mittal M et al (2019) New shape descriptor in the context of edge continuity, CAAI Transactions on Intelligence Technology 4(2)

  56. Tingting Y, Junqian W, Lintai W et al (2019) Three–stage network for age estimation, CAAI Transactions on Intelligence Technology 4(2)

  57. Vig E, Dorr M, Cox D (2012) Space–variant descriptor sampling for action recognition based on saliency and eye movements. European Conference of Computer Vision. https://doi.org/10.1007/978--3--642--33786--4

  58. Wang Y, Cang S, Yu H (2018) A Data Fusion–Based Hybrid Sensory System for Older People’s Daily Activity and Daily Routine Recognition. IEEE Sens J 18:6874–6888

  59. Wiens T (2019) Engine speed reduction for hydraulic machinery using predictive algorithms. Int J Hydromechatron 2(1):16–31

  60. Xia L, Chen CC, Aggarwal JK (2012) View invariant human action recognition using histograms of 3D joints. Proceedings of CVPRW. https://doi.org/10.1109/CVPRW.2012.6239233

  61. Xie C, Li C, Zhang B et al Memory Attention Networks for Skeleton-based Action Recognition. arXiv:https://arxiv.org/abs/1804.08254v2

  62. Yue H, Chen W (2015) Comments on Automatic Visual Bag–of–Words for Online Robot Navigation and Mapping. IEEE Transactions on Robotics 31:223–224

  63. Zhang B, Yang Y, Chen et al (2017) Action Recognition Using 3D Histograms of Texture and A Multi–Class Boosting Classifier. IEEE transactions on image processing 26(10). https://doi.org/10.1109/TIP.2017.2718189

  64. Zhang J, Shum H, Han J et al (2018) Action Recognition From Arbitrary Views Using Transferable Dictionary Learning. IEEE transactions on image processing 27(10). https://doi.org/10.1109/TIP.2018.2836323

  65. Zhu C, Miao D (2019) Influence of kernel clustering on an RBFN, CAAI Transactions on Intelligence Technology 4(4)

Download references

Acknowledgements

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (No. 2018R1D1A1A02085645) and by a grant (19CTAP-C152247-01) from Technology Advancement Research Program funded by Ministry of Land, Infrastructure and Transport of Korean government.

Author information

Authors and Affiliations

  1. Air University, E-9, Islamabad, Pakistan

    Amir Nadeem & Ahmad Jalal

  2. Department of Human-Computer Interaction, Hanyang University, Hanyang, South Korea

    Kibum Kim

Authors
  1. Amir Nadeem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmad Jalal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kibum Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kibum Kim.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nadeem, A., Jalal, A. & Kim, K. Automatic human posture estimation for sport activity recognition with robust body parts detection and entropy markov model. Multimed Tools Appl 80, 21465–21498 (2021). https://doi.org/10.1007/s11042-021-10687-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10687-5

Keywords

Search

Navigation