Skip to main content

Advertisement

SpringerLink
Log in

Automatic multi-gait recognition using pedestrian’s spatiotemporal features

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

This paper presents an automatic technique to detect and track the multiple pedestrians for their identifications in a video sequence. Contrarily to the most existing approaches, the proposed technique does not require human silhouette segmentation from the video to build the gait representation. Additionally, it also does not need to estimate the gait cycle to compute the gait-related features. The proposed technique comprises on four steps. In the first step, the pedestrian information is detected and tracked in the temporal direction. Second, we computed spatiotemporal features in the localized/tracked area to encode their walking patterns using dense trajectories. In the third step, the local features of pedestrian’s walk are transformed into its compact and high-level representation using Fisher vector encoding scheme. Fourth, these high-level representations are fed to simple linear support vector machine for the identification. Since there is no publicly available multi-subject gait dataset and the recording of a new dataset is an expensive process which also demands a long time, we generated an augmented gait dataset where multiple subjects are available in a video sequence to cope with this limitation. We employed the single-subject CASIA-B gait dataset to generate the augmented multi-subject gait video sequences. The identification of multiple pedestrians in the constructed augmented gait sequences is a challenging task as multiple subjects are walking beside and crossing each other, hence producing several types of occlusions. The proposed gait recognition algorithm achieved a recognition rate of 86.3% on multi-subject gait dataset and 98.6% on the single-subject gait dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

The data used in this study are openly available as CASIA-B dataset and can be found at http://www.cbsr.ia.ac.cn/GaitDatasetB-silh.zip.

Code Availability

The proposed multi-gait CASIA-B dataset is available at http://faculty.pucit.edu.pk/~farid/Research/multiGait.html.

Notes

  1. http://faculty.pucit.edu.pk/~farid/Research/multiGait.html.

  2. http://www.am.sanken.osaka-u.ac.jp/BiometricDB/dataset/GaitLP/Benchmarks.html.

References

  1. Khan MH (2018) Human activity analysis in visual surveillance and healthcare, vol 45. Logos Verlag Berlin GmbH, Berlin

    Google Scholar 

  2. Bouchrika I, Nixon MS (2007) Model-based feature extraction for gait analysis and recognition. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Springer, pp 150–160

  3. Wang L, Ning H, Tan T, Hu W (2004) Fusion of static and dynamic body biometrics for gait recognition. IEEE Trans Circuits Syst Video Technol 14(2):149–158

    Google Scholar 

  4. Han J, Bhanu B (2006) Individual recognition using gait energy image. IEEE Trans Pattern Anal Mach Intell 28(2):316–322

    Google Scholar 

  5. Zeng W, Wang C, Yang F (2014) Silhouette-based gait recognition via deterministic learning. Pattern Recognit 47(11):3568–3584

    Google Scholar 

  6. Chao H, He Y, Zhang J, Feng J (2019) Gaitset: regarding gait as a set for cross-view gait recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 33, pp 8126–8133

  7. Khan MH, Farid MS, Grzegorzek M (2020) A non-linear view transformations model for cross-view gait recognition. Neurocomputing 402:100-111

  8. Yang Y, Tu D, Li G (2014) Gait recognition using flow histogram energy image. In: Proceedings of 13th International Conference on Pattern Recognition (ICPR), pp 444–449

  9. Ariyanto G, Nixon MS (2012) Marionette mass-spring model for 3D gait biometrics. In: IEEE International Conference Biometrics. IEEE, pp 354–359

  10. Wang L, Tan T, Hu W, Ning H et al (2003) Automatic gait recognition based on statistical shape analysis. IEEE Trans Image Process 12(9):1120–1131

    MathSciNet  Google Scholar 

  11. Tan D, Huang K, Yu S, Tan T (2007) Uniprojective features for gait recognition. In: Proceedings of the International Joint Conference on Biometrics. Springer, pp 673–682

  12. Khan MH, Schneider M, Farid MS, Grzegorzek M (2018) Detection of infantile movement disorders in video data using deformable part-based model. Sensors 18:3202

    Google Scholar 

  13. Kusakunniran W (2014) Attribute-based learning for gait recognition using spatio-temporal interest points. Image Vis Comput 32(12):1117–1126

    Google Scholar 

  14. Wu Z, Huang Y, Wang L, Wang X, Tan T (2016) A comprehensive study on cross-view gait based human identification with deep CNNs. IEEE Trans Pattern Anal Mach Intell 39(2):209–226

    Google Scholar 

  15. Nair BM, Kendricks KD (2016) Deep network for analyzing gait patterns in low resolution video towards threat identification. Electron Imaging 2016(11):1–8

    Google Scholar 

  16. Liu D, Ye M, Li X, Zhang F, Lin L (2016) Memory-based gait recognition. In: BMVC, pp 1–12

  17. Wang Y, Song C, Huang Y, Wang Z, Wang L (2019) Learning view invariant gait features with two-stream GAN. Neurocomputing 339:245–254

    Google Scholar 

  18. Batchuluun G, Yoon HS, Kang JK, Park KR (2018) Gait-based human identification by combining shallow convolutional neural network-stacked long short-term memory and deep convolutional neural network. IEEE Access 6:63164–63186

    Google Scholar 

  19. Zhang Z, Tran L, Liu F, Liu X (2020) On learning disentangled representations for gait recognition. IEEE Trans Pattern Anal Mach Intell 44(1):345–360

    Google Scholar 

  20. Zhang Z, Tran L, Yin X, Atoum Y, Liu X, Wan J et al (2019) Gait recognition via disentangled representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 4710–4719

  21. Ortells J, Mollineda RA, Mederos B, Martín-Félez R (2017) Gait recognition from corrupted silhouettes: a robust statistical approach. Mach Vis Appl 28(1–2):15–33

    Google Scholar 

  22. Roy A, Chattopadhyay P, Sural S, Mukherjee J, Rigoll G (2015) Modelling, synthesis and characterisation of occlusion in videos. IET Comput Vis 9(6):821–830

    Google Scholar 

  23. Hofmann M, Sural S, Rigoll G (2011) Gait recognition in the presence of occlusion: a new dataset and baseline algorithms. Václav Skala-UNION Agency

  24. Singh JP, Arora S, Jain S, SoM UPS (2019) A multi-gait dataset for human recognition under occlusion scenario. In: International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT). IEEE, vol 1, pp 1–6

  25. Yu S et al (2006) A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. In: Proceedings of International Conference on Pattern Recognition (ICPR), vol 4, pp 441–444

  26. Khan MH, Farid MS, Grzegorzek M (2018) Spatiotemporal feature of human motion for gait recognition. Signal Image Video Process 13:369–377

    Google Scholar 

  27. Lee L, Grimson WEL (2002) Gait analysis for recognition and classification. In: International Conference on Automatic Face and Gesture Recognition. IEEE, pp 155–162

  28. Tafazzoli F, Safabakhsh R (2010) Model-based human gait recognition using leg and arm movements. Eng Appl Artif Intell 23(8):1237–1246

    Google Scholar 

  29. Chai Y, Wang Q, Jia J, Zhao R (2006) A novel human gait recognition method by segmenting and extracting the region variance feature. In: Proceedings of International Conference on Pattern Recognition (ICPR), vol 4, pp 425–428

  30. Yoo JH, Hwang D, Moon KY, Nixon MS (2008) Automated human recognition by gait using neural network. In: 1st Workshops on Image Processing Theory, Tools and Applications. IEEE, pp 1–6

  31. Yoo JH, Nixon MS (2011) Automated markerless analysis of human gait motion for recognition and classification. ETRI J 33(2):259–266

    Google Scholar 

  32. Yam C, Nixon MS, Carter JN (2004) Automated person recognition by walking and running via model-based approaches. Pattern Recognit 37(5):1057–1072

    Google Scholar 

  33. Lu W, Zong W, Xing W, Bao E (2014) Gait recognition based on joint distribution of motion angles. J Vis Lang Comput 25(6):754–763

    Google Scholar 

  34. Khan MH, Farid MS, Grzegorzek M (2021) Vision-based approaches towards person identification using gait. Comput Sci Rev 42:100432

    MathSciNet  Google Scholar 

  35. Wang C, Zhang J, Wang L, Pu J, Yuan X (2012) Human identification using temporal information preserving gait template. IEEE Trans Pattern Anal Mach Intell 34(11):2164–2176

    Google Scholar 

  36. Arora P, Hanmandlu M, Srivastava S (2015) Gait based authentication using gait information image features. Pattern Recognit Lett 68:336–342

    Google Scholar 

  37. Aqmar MR, Fujihara Y, Makihara Y, Yagi Y (2014) Gait recognition by fluctuations. Comput Vis Image Underst 126:38–52

    Google Scholar 

  38. Yang X, Zhou Y, Zhang T, Shu G, Yang J (2008) Gait recognition based on dynamic region analysis. Signal Process 88(9):2350–2356

    MATH  Google Scholar 

  39. Luo J, Zhang J, Zi C, Niu Y, Tian H, Xiu C (2015) Gait recognition using GEI and AFDEI. Int J Opt. https://doi.org/10.1155/2015/763908

    Article  Google Scholar 

  40. Zhang E, Zhao Y, Xiong W (2010) Active energy image plus 2DLPP for gait recognition. Signal Process 90(7):2295–2302

    MATH  Google Scholar 

  41. Bukhari M, Durrani MY, Gillani S, Yasmin S, Rho S, Yeo SS (2022) Exploiting vulnerability of convolutional neural network-based gait recognition system. J Supercomput 78(17):18578–18597

    Google Scholar 

  42. Goffredo M, Carter JN, Nixon MS (2008) Front-view gait recognition. In: IEEE International Conference on Biometrics: Theory, Applications, and Systems (BTAS) (BTAS). IEEE, pp 1–6

  43. Shaban Al-Ani M, Mohammadi M, AlyanNezhadi M (2020) Gait recognition based on measurements of moving human legs angles. Int J Eng 33(5):975–983

    Google Scholar 

  44. Castro FM, Marín-Jiménez MJ, Guil N (2016) Multimodal features fusion for gait, gender and shoes recognition. Mach Vis Appl 27(8):1213–28

    Google Scholar 

  45. Jeong S, Kim Th, Cho J (2013) Gait recognition using description of shape synthesized by planar homography. J Supercomput 65(1):122–135

    Google Scholar 

  46. Wang L, Tan T, Ning H, Hu W (2003) Silhouette analysis-based gait recognition for human identification. IEEE Trans Pattern Anal Mach Intell 25(12):1505–1518

    Google Scholar 

  47. Dadashi F, Araabi BN, Soltanian-Zadeh H (2009) Gait recognition using wavelet packet silhouette representation and transductive support vector machines. In: IEEE International Congress on Image and Signal Processing (CISP); pp 1–5

  48. Castro FM, Marín-Jiménez MJ, Guil N, López-Tapia S, de la Blanca NP (2017) Evaluation of CNN architectures for gait recognition based on optical flow maps. In: International Conference of the Biometrics Special Interest Group (BIOSIG). IEEE, pp 1–5

  49. Sokolova A, Konushin A (2017) Gait recognition based on convolutional neural networks. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol 42

  50. Khan MH, Farid MS, Grzegorzek M (2017) Person identification using spatiotemporal motion characteristics. In: Proceedings of International Conference on Image Processing (ICIP). IEEE, pp 166–170

  51. Sheng W, Li X (2020) Siamese denoising autoencoders for joints trajectories reconstruction and robust gait recognition. Neurocomputing 365:86–94

    Google Scholar 

  52. Khan MH, Farid MS, Grzegorzek M (2019) A generic codebook based approach for gait recognition. Multimed Tools Appl 78(24):35689–35712

    Google Scholar 

  53. Wolf T, Babaee M, Rigoll G (2016) Multi-view gait recognition using 3D convolutional neural networks. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, pp 4165–4169

  54. Delgado-Escaño R, Castro FM, Cózar JR, Marín-Jiménez MJ, Guil N (2020) MuPeG—the multiple person gait framework. Sensors 20(5):1358

    Google Scholar 

  55. Sepas-Moghaddam A, Etemad A (2022) Deep gait recognition: a survey. IEEE Trans Pattern Anal Mach Intell 45(1):264–284

    Google Scholar 

  56. Sánchez J, Perronnin F, Mensink T, Verbeek J (2013) Image classification with the fisher vector: theory and practice. Int J Comput Vis 105(3):222–245

    MathSciNet  MATH  Google Scholar 

  57. Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) LIBLINEAR: a library for large linear classification. J Mach Learn Res 9:1871–1874

    MATH  Google Scholar 

  58. Fan Q, Zhang L (2018) A novel patch matching algorithm for exemplar-based image inpainting. Multimed Tools Appl 77(9):10807–10821

    Google Scholar 

  59. Newson A, Almansa A, Fradet M, Gousseau Y, Pérez P (2014) Video inpainting of complex scenes. SIAM J Imag Sci 7(4):1993–2019

    MathSciNet  MATH  Google Scholar 

  60. Wang H, Schmid C (2013) Action recognition with improved trajectories. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp 3551–3558

  61. Khan MH, Farid MS, Grzegorzek M (2023) A comprehensive study on codebook-based feature fusion for gait recognition. Inf Fusion 92:216–230

    Google Scholar 

  62. Sánchez J, Perronnin F, Mensink T, Verbeek J (2013) Image classification with the fisher vector: theory and practice. Int J Comput Vis 105(3):222–245

    MathSciNet  MATH  Google Scholar 

  63. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B Stat Methodol 39:1–38

    MathSciNet  MATH  Google Scholar 

  64. Perronnin F, Sánchez J, Mensink T (2010) Improving the fisher kernel for large-scale image classification. Springer, Berlin, pp 143–156

    Google Scholar 

  65. Peng X, Wang L, Wang X, Qiao Y (2016) Bag of visual words and fusion methods for action recognition: comprehensive study and good practice. Comput Vis Image Underst 150:109–125

    Google Scholar 

  66. Khan MH, Li F, Farid MS, Grzegorzek M (2017) Gait recognition using motion trajectory analysis. In: Proceedings of the International Conference on Computer Recognition Systems(CORES). Springer, pp 73–82

  67. Jaakkola T, Haussler D (1999) Exploiting generative models in discriminative classifiers. In: AAdvances in Neural Information Processing Systems, pp 487–493

  68. Cheng G, Yang J, Gao D, Guo L, Han J (2020) High-quality proposals for weakly supervised object detection. IEEE Trans Image Process 29:5794–5804

    MATH  Google Scholar 

  69. Iwama H, Okumura M, Makihara Y, Yagi Y (2012) The OU-ISIR gait database comprising the large population dataset and performance evaluation of gait recognition. IEEE Trans Inf Forensics Secur 7(5):1511–1521

    Google Scholar 

  70. Li C, Min X, Sun S, Lin W, Tang Z (2017) Deepgait: a learning deep convolutional representation for view-invariant gait recognition using joint Bayesian. Appl Sci 7(3):210

    Google Scholar 

  71. Shiraga K, Makihara Y, Muramatsu D, Echigo T, Yagi Y (2016) Geinet: view-invariant gait recognition using a convolutional neural network. In: IEEE International Biometric Conferences. IEEE, pp 1–8

  72. Chen Q, Wang Y, Liu Z, Liu Q, Huang D (2017) Feature Map Pooling for Cross-View Gait Recognition Based on Silhouette Sequence Images. arXiv preprint arXiv:1711.09358

  73. Wu Z, Huang Y, Wang L, Wang X, Tan T (2017) A comprehensive study on cross-view gait based human identification with deep CNNs. IEEE Trans Pattern Anal Mach Intell 39(2):209–226

    Google Scholar 

  74. Wang J, Peng K (2020) A Multi-View Gait Recognition Method Using Deep Convolutional Neural Network and Channel Attention Mechanism. Computer Modeling in Engineering & Sciences. 125(1):345–363

    Google Scholar 

  75. Wu H, Tian J, Fu Y, Li B, Li X (2020) Condition-aware comparison scheme for gait recognition. IEEE Trans Image Process 30:2734–2744

    Google Scholar 

  76. Işık SG, Ekenel HK (2021) Deep convolutional feature-based gait recognition using silhouettes and RGB images. In: 2021 6th International Conference on Computer Science and Engineering (UBMK). IEEE, pp 336–341

  77. Qin H, Chen Z, Guo Q, Wu QJ, Lu M (2021) RPNet: gait recognition with relationships between each body-parts. IEEE Trans Circuits Syst Video Technol 32(5):2990–3000

    Google Scholar 

  78. Xiao J, Yang H, Xie K, Zhu J, Zhang J (2022) Learning discriminative representation with global and fine-grained features for cross-view gait recognition. CAAI Trans Intell Technol 7(2):187–199

    Google Scholar 

  79. Huang T, Ben X, Gong C, Zhang B, Yan R, Wu Q (2022) Enhanced spatial-temporal salience for cross-view gait recognition. IEEE Trans Circuits Syst Video Technol 32(10):6967–6980

    Google Scholar 

Download references

Funding

This research received no funding.

Author information

Author notes

  1. Muhammad Hassan Khan and Hiba Azam contributed equally to this work.

Authors and Affiliations

  1. Faculty of Computing & Information Technology, University of the Punjab, Lahore, Punjab, 54500, Pakistan

    Muhammad Hassan Khan, Hiba Azam & Muhammad Shahid Farid

Authors
  1. Muhammad Hassan Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hiba Azam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muhammad Shahid Farid
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MHK and HA conceived the idea. MHK, MSF, and HA designed the algorithm. HA and MHK carried out experiments and wrote the main manuscript text. HA, MSF., and MHK did the validation and discussion. MHK, and MSF supervised the research and proofread the manuscript.

Corresponding author

Correspondence to Muhammad Hassan Khan.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, M.H., Azam, H. & Farid, M.S. Automatic multi-gait recognition using pedestrian’s spatiotemporal features. J Supercomput 79, 19254–19276 (2023). https://doi.org/10.1007/s11227-023-05391-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05391-0

Keywords

Search

Navigation