Skip to main content
Springer Nature Link
Log in

Cross-view gait recognition based on residual long short-term memory

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

As a promising biometric recognition technology, gait recognition has many advantages, such as non-invasive, easy to implement in a long distance, but it is very sensitive to the change of video acquisition angles. In this paper, we propose a novel cross-view gait recognition framework based on residual long short-term memory, namely, CVGR-RLSTM, to extract intrinsic gait features and carry out gait recognition. The proposed framework captures dependencies of human postures in time dimension during walking by inputting randomly sampling frame-by-frame gait energy images. The frame-by-frame gait energy images are generated by merging adjacent gait silhouette images sequentially, which integrates gait features of temporal and spatial dimensions to a certain extent. In the CVGR-RLSTM framework, the embedded residual module is used to further refine the spatial gait features, and the LSTM module is utilized to optimize the temporal gait features. To evaluate the proposed framework, we carried out a series of comparative experiments on the CASIA Dataset B and OU-ISIR LP Dataset. Experimental results show that the proposed method reaches the state-of-the-art level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Abdulsattar F, Carter J (2016) Performance analysis of gait recognition with large perspective distortion. In: IEEE International Conference on Identity, Security and Behaviour Analysis. Sendai, Japan, pp 1–6

  2. Battistone F, Petrosino A (2019) Tglstm: A time based graph deep learning approach to gait recognition. Pattern Recogn Lett 126:132–138

    Article  Google Scholar 

  3. Boulgouris NV, Huang X (2013) Gait recognition using hmms and dual discriminative observations for sub-dynamics analysis. IEEE Trans Image Process 22(9):3636–3647

    Article  Google Scholar 

  4. Deng M, Wang C, Cheng F, Zeng W (2017) Fusion of spatial-temporal and kinematic features for gait recognition with deterministic learning. Pattern Recogn 67:186–200

    Article  Google Scholar 

  5. Ercolano G, Rossi S (2021) Combining cnn and lstm for activity of daily living recognition with a 3d matrix skeleton representation. Intel Serv Robot 14:1–11

    Article  Google Scholar 

  6. Feng Y, Li Y, Luo J (2016) Learning effective gait features using lstm. In: 23rd International conference on pattern recognition. Cancun, pp 325–330

  7. Gao Z, Nie W, Liu A, Zhang H (2016) Evaluation of local spatial-temporal features for cross-view action recognition. Neurocomput. 173(P1):110–117

    Article  Google Scholar 

  8. Gao Z, Xuan H, Zhang H, Wan S, Choo KR (2019) Adaptive fusion and category-level dictionary learning model for multiview human action recognition. IEEE Internet of Things Journal 6(6):9280–9293

    Article  Google Scholar 

  9. Han J, BirBhanu (2006) Individual recognition using gait energy image. IEEE Trans Pattern Anal Mach Intell 28(2):316–322

    Article  Google Scholar 

  10. Han J, Pauwels E, Zeeuw P (2012) Employing a rgb-d sensor for real-time tracking of humans across multiple re-entries in a smart environment. IEEE Trans Consum Electron 58:255–263

    Article  Google Scholar 

  11. He K, Zhang X, Ren S (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition, pp 770–778. Las Vegas, NV

  12. Hong C, Yu J, Wan J, Tao D, Wang M (2015) Multimodal deep autoencoder for human pose recovery. IEEE Trans Image Process 24(12):5659–5670

    Article  MathSciNet  Google Scholar 

  13. Hong C, Yu J, Zhang J, Jin X, Lee K (2019) Multimodal face-pose estimation with multitask manifold deep learning. IEEE Transactions on Industrial Informatics 15(7):3952–3961

    Article  Google Scholar 

  14. Iwama H, Okumura M, Makihara Y, Yagi Y (2012) The ou-isir gait database comprising the large population dataset and performance evaluation of gait recognition. IEEE Trans Inf Forensics Secur 7(5):1511–1521

    Article  Google Scholar 

  15. Kai C, Qiang H (2016) Training deep bidirectional lstm acoustic model for lvcsr by a context-sensitive-chunk bptt approach. IEEE-ACM Transactions on Audio Speech and Language Processing 24(7):1185–1193

    Article  Google Scholar 

  16. Le Cun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(5):436–445

    Google Scholar 

  17. Muramatsu D, Shiraishi A, Makihara Y, Uddin MZ, Yagi Y (2015) Gait-based person recognition using arbitrary view transformation model. IEEE Trans Image Process 24(1):140–154

    Article  MathSciNet  Google Scholar 

  18. Muramatsu D, Makihara Y, Yagi Y (2016) View transformation model incorporating quality measures for cross-view gait recognition. IEEE Trans Cybern 46(7):1602–1615

    Article  Google Scholar 

  19. Perrett T, Damen D (2019) Ddlstm: Dual-domain lstm for cross-dataset action recognition, pp 7844–7853

  20. Sarkar S, Phillips PJ, Liu Z, Vega IR, Grother P, Bowyer KW (2005) The humanid gait challenge problem: data sets, performance, and analysis. IEEE Trans Pattern Anal Mach Intell 27(2):162–177

    Article  Google Scholar 

  21. Shiraga K, Makihara Y, Muramatsu D, Echigo T, Y. Yagi (2016) VGeinet: View-invariant gait recognition using a convolutional neural network. In: International conference on biometrics. Halmstad, pp 1–8

  22. Takemura N, Makihara Y, Muramatsu D, Echigo T, Yagi Y (2018) On input/output architectures for convolutional neural network-based cross-view gait recognition. IEEE Trans Circuits Syst Video Technol 28(1):1–13

    Article  Google Scholar 

  23. Tang J, Luo J, Tjahjadi T, Guo F (2017) Robust arbitrary-view gait recognition based on 3d partial similarity matching. IEEE Trans Image Process 26(1):7–22

    Article  MathSciNet  Google Scholar 

  24. Thapar D, Nigam A, Aggarwal D, Agarwal P (2018) Vgr-net: A view invariant gait recognition network. In: IEEE 4th International conference on identity, security, and behavior analysis. Singapore, pp 1–8

  25. Wang X, Yan WQ (2020) Cross-view gait recognition through ensemble learning. Neural Comput and Applic 32(11):7275–7287

    Article  Google Scholar 

  26. Wang X, Wang J, Yan K (2018) Gait recognition based on gabor wavelets and (2D)2PCA. Multimed Tools Appl 77(10):12545–12561

    Article  Google Scholar 

  27. Wang X, Feng S, Yan WQ (2019) Human gait recognition based on self-adaptive hidden markov model. IEEE Transactions on Computational Biology and Bioinformatics, pp 1–12

  28. Wang X, Zhang J, Yan WQ (2019) Gait recognition using multichannel convolution neural networks. Neural Computing and Applications

  29. Wang X, Yan WQ (2020) Human gait recognition based on frame-by-frame gait energy images and convolutional long short term memory. Int J Neural Sys 30(1):1950027

    Article  Google Scholar 

  30. Watanabe Y, Kimura M (2020) Gait identification and authentication using lstm based on 3-axis accelerations of smartphone. Procedia Computer Science 176:3873–3880

    Article  Google Scholar 

  31. Wolf T, Babaee M, Rigoll G (2016) Multi-view gait recognition using 3d convolutional neural networks. In: IEEE International conference on image processing. Phoenix, AZ, pp 4165–4169

  32. Wu Z, Huang Y, Wang L, Wang X, Tan T (2017) A comprehensive study on cross-view gait based human identification with deep cnns. IEEE Trans Pattern Anal Mach Intell 39(2):209–226

    Article  Google Scholar 

  33. Yanghao L, Naiyan W, Jianping S, Xiaodi H, Jiaying L (2018) Adaptive batch normalization for practical domain adaptation. Pattern Recognition

  34. Yu S, Chen H, Reyes EBG, Poh N (2017) Gaitgan: Invariant gait feature extraction using generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, HI, pp 532–539

  35. Yu J, Tan M, Zhang H, Tao D, Rui Y (2019) Hierarchical deep click feature prediction for fine-grained image recognition. IEEE Trans Pattern Anal Mach Intell, pp 1–1

  36. Yu S, Chen H, Wang Q, Shen L, Huang Y (2017) Invariant feature extraction for gait recognition using only one uniform model. Neurocomputing 239:81–93

    Article  Google Scholar 

  37. Yu S, Tan D, Tan T (2006) A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. In: International conference on pattern recognition, pp 441–444. Hong Kong, China

Download references

Acknowledgements

This research was funded by the National Natural Science Foundation of China, grant number No. 61602431 and the Natural Science Foundation of Zhejiang Province, grant number No.Y20F020113.

Author information

Authors and Affiliations

  1. Zhejiang Technical Institute of Economics, Hangzhou, 310058, China

    Junqin Wen

  2. Key Laboratory of Electromagnetic Wave Information Technology and Metrology of Zhejiang Province, College of Information Engineering, China Jiliang University, Hangzhou, 310018, China

    Xiuhui Wang

Authors
  1. Junqin Wen
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Xiuhui Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Xiuhui Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, J., Wang, X. Cross-view gait recognition based on residual long short-term memory. Multimed Tools Appl 80, 28777–28788 (2021). https://doi.org/10.1007/s11042-021-11107-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11107-4

Keywords