Skip to main content
SpringerLink
Log in

An implementation of optimized framework for action classification using multilayers neural network on selected fused features

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

In video sequences, human action recognition is a challenging problem due to motion variation, in frame person difference, and setting of video recording in the field of computer vision. Since last few years, applications of human activity recognition have increased significantly. In the literature, many techniques are implemented for human action recognition, but still they face problem in contrast of foreground region, segmentation, feature extraction, and feature selection. This article contributes a novel human action recognition method by embedding the proposed frames fusion working on the principle of pixels similarity. An improved hybrid feature extraction increases the recognition rate and allows efficient classification in the complex environment. The design consists of four phases, (a) enhancement of video frames (b) threshold-based background subtraction and construction of saliency map (c) feature extraction and selection (d) neural network (NN) for human action classification. Results have been tested using five benchmark datasets including Weizmann, KTH, UIUC, Muhavi, and WVU and obtaining recognition rate 97.2, 99.8, 99.4, 99.9, and 99.9%, respectively. Contingency table and graphical curves support our claims. Comparison with existent techniques identifies the recognition rate and trueness of our proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Ye Q, Han Z, Jiao J, Liu J (2013) Human detection in images via piecewise linear support vector machines. IEEE Trans Image Process 22(2):778–789

    Article  MathSciNet  MATH  Google Scholar 

  2. Liang C-W, Juang C-F (2015) Moving object classification using local shape and HOG features in wavelet-transformed space with hierarchical SVM classifiers. Appl Soft Comput 28:483–497

    Article  Google Scholar 

  3. Xu Y, Dong X, Lin S, Han TX, Cao X, Li X (2012) Detection of sudden pedestrian crossings for driving assistance systems. IEEE Trans Syst Man Cybern B Cybern 42(3):729–739

    Article  Google Scholar 

  4. Juang C-F, Chang C-M (2007) Human body posture classification by a neural fuzzy network and home care system application. IEEE Trans Syst Man Cybern Part A Syst Hum 37(6):984–994

    Article  Google Scholar 

  5. Ye Q, Liang J, Jiao J (2012) Pedestrian detection in video images via error correcting output code classification of manifold subclasses. IEEE Trans Intell Transp Syst 13(1):193–202

    Article  Google Scholar 

  6. Yeguas-Bolivar E, Muoz-Salinas R, Medina-Carnicer R, Carmona-Poyato A (2014) Comparing evolutionary algorithms and particle filters for Markerless Human Motion Capture. Appl Soft Comput 17:153–166

    Article  Google Scholar 

  7. Aggarwal JK, Ryoo MS (2011) Human activity analysis: a review. ACM Comput Surv (CSUR) 43(3):16

    Article  Google Scholar 

  8. Chathuramali KGM, Rodrigo R (2013) Faster human activity recognition with SVM. In: IEEE international conference on advances in ICT for emerging regions, pp 197–203

  9. Ali S, Shah M (2010) Human action recognition in videos using kinematic features and multiple instance learning. IEEE Trans Pattern Anal Mach Intell 32(2):288–303

    Article  Google Scholar 

  10. Dayananda Kumar NC, Suresh KV (2014) HOG-PCA descriptor with optical flow based human detection and tracking. In: International conference on communications and signal processing (ICCSP). IEEE, pp 900–904

  11. Porikli F, Tuzel O, Meer P (2016) Designing a boosted classifier on riemannian manifolds. In: Riemannian computing in computer vision. Springer, pp 281–301

  12. Mu Y, Yan S, Liu Y, Huang T, Zhou B (2008) Discriminative local binary patterns for human detection in personal album. In: Proceedings of IEEE international conference computer vision pattern recognition, pp 1–8

  13. Wang X, Han TX, Yan S (2009) An HOG-LBP human detector with partial occlusion handling. In: Proceedings of IEEE international conference on computer vision, pp 32–39

  14. Nguyen DT, Ogunbona PO, Li W (2013) A novel shape-based non-redundant local binary pattern descriptor for object detection. Pattern Recognit 46(5):1485–1500

    Article  Google Scholar 

  15. Xia L, Chen C-C, Aggarwal JK (2011) Human detection using depth information by kinect. In: 2011 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 15–22

  16. Kramer O (2013) K-nearest neighbors. In: Dimensionality reduction with unsupervised nearest neighbors. Springer, Berlin, pp 13–23

  17. Shen H-B, Chou K-C (2006) Ensemble classifier for protein fold pattern recognition. Bioinformatics 22(14):1717–1722

    Article  Google Scholar 

  18. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of IEEE international conference on computer vision pattern recognition, pp 886–893

  19. Dalal N, Triggs B, Schmid C (2006) Human detection using oriented histograms of flow and appearance. In: European conference on computer vision, Springer

  20. Tran D, Sorokin A, Forsyth D (2008) Human activity recognition with metric learning, Computer Vision – ECCV 2008, pp 548-561

  21. Gilbert A, Illingworth J, Bowden R (2009) Fast realistic multi-action recognition using mined dense spatio-temporal features. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 925–931

  22. Bregonzio M, Gong S, Xiang T (2009) Recognising action as clouds of space-time interest points. In: IEEE conference on computer vision and pattern recognition, 2009, CVPR. IEEE, pp 1948–1955

  23. Yang M, Lv F, Xu W, Yu K, Gong Y (2009) Human action detection by boosting efficient motion features. In: 2009 IEEE 12th international conference on computer vision workshops (ICCV Workshops). IEEE, pp 522–529

  24. Liu L, Shao L, Rockett P (2013) Human action recognition based on boosted feature selection and naive Bayes nearest-neighbor classification. Signal Process 93(6):1521–1530

    Article  Google Scholar 

  25. Qian H, Mao Y, Xiang W, Wang Z (2010) Recognition of human activities using SVM multi-class classifier. Pattern Recognit Lett 31(2):100–111

    Article  Google Scholar 

  26. Mahbub U, Imtiaz H, Ahad MAR (2012) Motion clustering-based action recognition technique using optical flow. In: 2012 International conference on informatics, electronics & vision (ICIEV). IEEE, pp 919–924

  27. Varol G, Laptev I, Schmid C (2017) Long-term temporal convolutions for action recognition. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2017.2712608

  28. Zhang H, Cao X, Ho JKL, Chow TWS (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inform 13(2):520–531

    Article  Google Scholar 

  29. Li C, Zhong Q, Xie D, Pu S (2017) Skeleton-based action recognition with convolutional neural networks. arXiv preprint arXiv:1704.07595

  30. Kushwaha AKS, Singh S, Srivastava R (2017) Multi-view human activity recognition based on silhouette and uniform rotation invariant local binary patterns. Multimed Syst 23(4):451–467

    Article  Google Scholar 

  31. Gonzalez RC, Woods RE (2002) Digital image processing. Prentice hall, New Jersey

    Google Scholar 

  32. Hunter RS (1948) Photoelectric color-difference meter. JOSA 38(7):661 (Proceedings of the Winter Meeting of the Optical Society of America)

    Google Scholar 

  33. Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of optical flow techniques. Int J Comput Vision 12(1):43–77

    Article  Google Scholar 

  34. Kim J, Han D, Tai Y-W, Kim J (2014) Salient region detection via high-dimensional color transform. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 883–890

  35. Kempe J, Regev O, Toner B (2010) Unique games with entangled provers are easy. SIAM J Comput 39(7):3207–3229

    Article  MathSciNet  MATH  Google Scholar 

  36. Ito T, Kobayashi H, Matsumoto K (2009) Oracularization and two-prover one-round interactive proofs against nonlocal strategies. In: 24th annual IEEE conference on computational complexity, 2009. CCC’09. IEEE, pp 217–228

  37. Duan Q, Akram T, Duan P, Wang X (2016) Visual saliency detection using information contents weighting. Optik-Int J Light Electron Opt 127(19):7418–7430

    Article  Google Scholar 

  38. Costa AF, Humpire-Mamani G, Traina AJM (2012) An efficient algorithm for fractal analysis of textures. In: 2012 25th SIBGRAPI conference on graphics, patterns and images (SIBGRAPI). IEEE, pp 39–46

  39. Xiong W, Zhang L, Bo D, Tao D (2017) Combining local and global: rich and robust feature pooling for visual recognition. Pattern Recognit 62:225–235

    Article  Google Scholar 

  40. Zhang L, Zhang Q, Zhang L, Tao D, Huang X, Bo D (2015) Ensemble manifold regularized sparse low-rank approximation for multiview feature embedding. Pattern Recognit 48(10):3102–3112

    Article  Google Scholar 

  41. Jolliffe I (2002) Principal component analysis. Wiley, Hoboken

    MATH  Google Scholar 

  42. Kamran M, Haider SA, Akram T, Naqvi SR, He SK (2016) Prediction of IV curves for a superconducting thin film using artificial neural networks. Superlattices Microstruct 95:88–94

    Article  Google Scholar 

  43. Baumann F, Ehlers A, Rosenhahn B, Liao J (2016) Recognizing human actions using novel space-time volume binary patterns. Neurocomputing 173:54–63

    Article  Google Scholar 

  44. Wang H, Kläser A, Schmid C, Liu C-L (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vision 103(1):60–79

    Article  MathSciNet  Google Scholar 

  45. Singh S, Velastin SA, Ragheb H (2010) Muhavi: a multicamera human action video dataset for the evaluation of action recognition methods. In: 2010 Seventh IEEE international conference on advanced video and signal based surveillance (AVSS). IEEE, pp 48–55

  46. Ramagiri S, Kavi R, Kulathumani V (2011) Real-time multi-view human action recognition using a wireless camera network. In: 2011 Fifth ACM/IEEE international conference on distributed smart cameras (ICDSC). IEEE, pp 1–6

  47. Maity S, Bhattacharjee D, Chakrabarti A (2017) A novel approach for human action recognition from silhouette images. IETE J Res 63(2):160–171

    Article  Google Scholar 

  48. Xiao Q, Song R (2017) Action recognition based on hierarchical dynamic Bayesian network. Multimedia Tools and Applications. NY, USA, New York, pp 1–14

    Google Scholar 

  49. Wang H, Yuan C, Weiming H, Sun C (2012) Supervised class-specific dictionary learning for sparse modeling in action recognition. Pattern Recognit 45(11):3902–3911

    Article  Google Scholar 

  50. Chaaraoui AA, Climent-Prez P, Flrez-Revuelta F (2013) Silhouette-based human action recognition using sequences of key poses. Pattern Recognit Lett 34(15):1799–1807

    Article  Google Scholar 

  51. Nasiri JA, Charkari NM, Mozafari K (2014) Energy-based model of least squares twin Support Vector Machines for human action recognition. Signal Process 104:248–257

    Article  Google Scholar 

  52. Ahmad T, Rafique J, Muazzam H, Rizvi T (2015) Using discrete cosine transform based features for human action recognition. J Image Gr 3(2)

  53. Kumar SS, John M (2016) Human activity recognition using optical flow based feature set. In: 2016 IEEE international carnahan conference on security technology (ICCST). IEEE, pp 1–5

  54. Laptev I, Marszalek M, Schmid C, Rozenfeld B (2008) Learning realistic human actions from movies. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008. IEEE, pp 1–8

  55. Gilbert A, Illingworth J, Bowden R (2011) Action recognition using mined hierarchical compound features. IEEE Trans Pattern Anal Mach Intell 33(5):883–897

    Article  Google Scholar 

  56. Abdul-Azim HA, Hemayed EE (2015) Human action recognition using trajectory-based representation. Egypt Inform J 16(2):187–198

    Article  Google Scholar 

  57. Shao L, Liu L, Mengyang Y (2016) Kernelized multiview projection for robust action recognition. Int J Comput Vision 118(2):115–129

    Article  MathSciNet  Google Scholar 

  58. Tong M, Wang H, Tian W, Yang S (2017) Action recognition new framework with robust 3D-TCCHOGAC and 3D-HOOFGAC. Multimed Tools Appl 76(2):3011–3030

    Article  Google Scholar 

  59. Liu J, Kuipers B, Savarese S (2011) Recognizing human actions by attributes. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 3337–3344

  60. Parikh D, Grauman K (2011) Relative attributes. In: 2011 International conference on computer vision. IEEE, pp 503–510

  61. Zhang Z, Wang C, Xiao B, Zhou W, Liu S (2015) Robust relative attributes for human action recognition. Pattern Anal Appl 18(1):157–171

    Article  MathSciNet  Google Scholar 

  62. Wu X, Jia Y (2012) View-invariant action recognition using latent kernelized structural SVM. In: European conference on computer vision. Springer, Berlin, pp 411–424

  63. Cheema S, Eweiwi A, Thurau C, Bauckhage C (2011) Action recognition by learning discriminative key poses. In: 2011 IEEE international conference on computer vision workshops (ICCV Workshops). IEEE, pp 1302–1309

  64. Zheng J, Jiang Z, Chellappa R (2016) Cross-view action recognition via transferable dictionary learning. IEEE Trans Image Process 25(6):2542–2556

    Article  MathSciNet  MATH  Google Scholar 

  65. Murtaza F, Yousaf MH, Velastin SA (2016) Multi-view human action recognition using 2D motion templates based on MHIs and their HOG description. IET Comput Vision 10(7):758–767

    Article  Google Scholar 

  66. Haq A, Gondal I, Murshed M (2013) On temporal order invariance for view-invariant action recognition. IEEE Trans Circuits Syst Video Technol 23(2):203–211

    Article  Google Scholar 

  67. Kavi R, Kulathumani V, Rohit F, Kecojevic V (2016) Multiview fusion for activity recognition using deep neural networks. J Electron Imaging 25(4):043010–043010

    Article  Google Scholar 

  68. Iosifidis A, Tefas A, Pitas I (2013) Multi-view action recognition based on action volumes, fuzzy distances and cluster discriminant analysis. Signal Process 93(6):1445–1457

    Article  Google Scholar 

  69. Hashemi SM, Rahmati M (2016) View-independent action recognition: a hybrid approach. Multimed Tools Appl 75(12):6755–6775

    Article  Google Scholar 

  70. Liu R, Gillies DF (2016) Overfitting in linear feature extraction for classification of high-dimensional image data. Pattern Recognit 53:73–86

    Article  MATH  Google Scholar 

  71. Wu D, Sharma N, Blumenstein M (2017) Recent advances in video-based human action recognition using deep learning: a review. In: 2017 International joint conference on neural networks (IJCNN). IEEE, pp 2865–2872

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, COMSATS Institute of Information Technology, Wah Cantt, Pakistan

    Muhammad Attique Khan, Muhammad Sharif & Mussarat Yasmin

  2. Department of EE, COMSATS Institute of Information Technology, Wah Cantt, Pakistan

    Tallha Akram

  3. Department of Computer Science & Engineering, HITEC University, Museum Road, Taxila, Pakistan

    Muhammad Attique Khan & Muhammad Younus Javed

  4. Department of Mathematics, COMSATS Institute of Information Technology, Wah Cantt, Pakistan

    Nazeer Muhammad

Authors
  1. Muhammad Attique Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tallha Akram
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muhammad Sharif
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Younus Javed
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nazeer Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mussarat Yasmin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Sharif.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, M.A., Akram, T., Sharif, M. et al. An implementation of optimized framework for action classification using multilayers neural network on selected fused features. Pattern Anal Applic 22, 1377–1397 (2019). https://doi.org/10.1007/s10044-018-0688-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-018-0688-1

Keywords

Search

Navigation