Skip to main content

Advertisement

SpringerLink
Log in

Gait Recognition in Different Terrains with IMUs Based on Attention Mechanism Feature Fusion Method

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Gait recognition is significant in the fields of disease diagnosis and rehabilitation training by studying the characteristics of human gait with different terrain. To address the problem that the transformation of different outdoor terrains can affect the gait of walkers, a gait recognition algorithm based on feature fusion with attention mechanism is proposed. First, the acceleration, angular velocity and angle information collected by the inertial measurement unit is used; then the acquired inertial gait data is divided into periods to obtain the period data of each step; then the features are extracted from the data, followed by the visualization of the one-dimensional data into two-dimensional images. A lightweight model is designed to combine convolutional neural network with attention mechanism, and a new attention mechanism-based feature fusion method is proposed in this paper for extracting features from multiple sensors and fusing them for gait recognition. The comparison experimental results show that the recognition accuracy of the model proposed in this paper can reach 89\(\%\), and it has good recognition effect on gait under different terrain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Sahu G, Parida P (2020) A contemporary survey on human gait recognition. J Inf Assur Secur 15(3):94–106

    Google Scholar 

  2. Guo Y, Deligianni F, Gu X, Yang GZ (2019) 3-D canonical pose estimation and abnormal gait recognition with a single RGB-D camera. IEEE Robot Autom Lett 4(4):3617–3624

    Article  Google Scholar 

  3. Verlekar TT, Soares LD, Correia PL (2018) Gait recognition in the wild using shadow silhouettes. Image Vis Comput 76:1–13

    Article  Google Scholar 

  4. Hong C, Yu J, Zhang J, Jin X, Lee KH (2018) Multimodal face-pose estimation with multitask manifold deep learning. IEEE Trans Ind Inf 15(7):3952–3961

    Article  Google Scholar 

  5. Hong C, Yu J, Wan J, Tao D, Wang M (2015) Multimodal deep autoencoder for human pose recovery. IEEE Trans Image Process 24:5659–5670

    Article  MathSciNet  MATH  Google Scholar 

  6. Gao H, Luo L, Pi M, Li Z, Li Q, Zhao K, Huang J (2019) EEG-based volitional control of prosthetic legs for walking in different terrains. IEEE Trans Autom Sci Eng 18(2):530–540

    Article  Google Scholar 

  7. Duan F, Lv Y, Sun Z, Li J (2022) Multi-scale learning for multimodal neurophysiological signals: gait pattern classification as an example. Neural Process Lett 54(3):2455–2470

    Article  Google Scholar 

  8. Joshi D (2021) Terrain-based gait recognition using EEG: comparing machine learning and deep learning models. In: 2021 international conference on computational performance evaluation (ComPE), pp 734–740

  9. Wang F, Yan L, Xiao J (2019) Human gait recognition system based on support vector machine algorithm and using wearable sensors. Sens Mater 31(4):1335–1349

    Google Scholar 

  10. Rani V, Kumar M (2023) Human gait recognition: A systematic review. Multimedia Tools Appl 1–35

  11. Yu S, Yang J, Huang TH, Zhu J, Visco CJ, Hameed F, Su H (2023) Artificial neural network-based activities classification, gait phase estimation, and prediction. Ann Biomedl Eng 1–14

  12. Aung MS, Thies SB, Kenney LP, Howard D, Selles RW, Findlow AH, Goulermas JY (2013) Automated detection of instantaneous gait events using time frequency analysis and manifold embedding. IEEE Trans Neural Syst Rehabil Eng 21(6):908–916

    Article  Google Scholar 

  13. Contreras-Cruz MA, Novo-Torres L, Villarreal DJ, Ramirez-Paredes JP (2023) Convolutional neural network and sensor fusion for obstacle classification in the context of powered prosthetic leg applications. Comput Electr Eng 108:108656

    Article  Google Scholar 

  14. Celik Y, Aslan MF, Sabanci K, Stuart S, Woo WL, Godfrey A (2022) Improving inertial sensor-based activity recognition in neurological populations. Sensors 22(24):9891

    Article  Google Scholar 

  15. Coelho RM, Gouveia J, Botto MA, Krebs HI, Martins J (2022) Real-time walking gait terrain classification from foot-mounted Inertial Measurement Unit using Convolutional Long Short-Term Memory neural network. Expert Syst Appl 203:117306

    Article  Google Scholar 

  16. Wang F, Wen S, Wu C, Zhang Y, Wang H (2011) Gait recognition based on the Fast Fourier Transform and SVM. In: 2011 Chinese control and decision conference (CCDC), pp 1091–1094

  17. Cha Y, Kim H, Kim D (2018) Flexible piezoelectric sensor-based gait recognition. Sensors 18(2):468

    Article  Google Scholar 

  18. Choi S, Youn IH, LeMay R, Burns S, Youn JH (2014) Biometric gait recognition based on wireless acceleration sensor using k-nearest neighbor classification. In: 2014 international conference on computing networking and communications (ICNC), pp 1091–1095

  19. Yu J, Li J, Yu Z, Huang Q (2019) Multimodal transformer with multi-view visual representation for image captioning. IEEE Trans Circuits Syst Video Technol 30(12):4467–4480

    Article  Google Scholar 

  20. Yu J, Tan M, Zhang H, Rui Y, Tao D (2019) Hierarchical deep click feature prediction for fine-grained image recognition. IEEE Trans Pattern Anal Mach Intell 44(2):563–578

    Article  Google Scholar 

  21. Zhang J, Cao Y, Wu Q (2021) Vector of locally and adaptively aggregated descriptors for image feature representation. Pattern Recogn 116:107952

    Article  Google Scholar 

  22. Zhang J, Yang J, Yu J, Fan J (2022) Semisupervised image classification by mutual learning of multiple self-supervised models. Int J Intell Syst 37(5):3117–3141

    Article  Google Scholar 

  23. Wen J, Shen Y, Yang J (2022) Multi-view gait recognition based on generative adversarial network. Neural Process Lett 54(3):1855–1877

    Article  Google Scholar 

  24. Zou Q, Wang Y, Wang Q, Zhao Y, Li Q (2020) Deep learning-based gait recognition using smartphones in the wild. IEEE Trans Inf Forensics Secur 15:3197–3212

    Article  Google Scholar 

  25. Negi S, CBS Negi P, Sharma S, Sharma N (2020) Electromyographic and acceleration signals-based gait phase analysis for multiple terrain classification using deep learning. Int J Adv Res Eng Technol 11(6):656–665

  26. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3146–3154

  27. Albahli S, Nida N, Irtaza A, Yousaf MH, Mahmood MT (2020) Melanoma lesion detection and segmentation using YOLOv4-DarkNet and active contour. IEEE Access 8:198403–198414

    Article  Google Scholar 

  28. Woo S, Park J, Lee JY, Kweon IS (2018) Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  29. Cao Q, Xu F, Li H (2022) User authentication by gait data from smartphone sensors using hybrid deep learning network. Mathematics 10(13):2283

    Article  Google Scholar 

  30. Sprager S, Juric MB (2015) Inertial sensor-based gait recognition: a review. Sensors 15(9):22089–22127

    Article  Google Scholar 

  31. Tao W, Chen H, Moniruzzaman M, Leu MC, Yi Z, Qin R (2021) Attention-based sensor fusion for human activity recognition using IMU signals. arXiv preprint arXiv:2112.11224

  32. Panebianco GP, Bisi MC, Stagni R, Fantozzi S (2018) Analysis of the performance of 17 algorithms from a systematic review: influence of sensor position, analysed variable and computational approach in gait timing estimation from IMU measurements. Gait Posture 66:76–82

    Article  Google Scholar 

  33. Lopez-Nava IH, Munoz-Melendez A (2016) Wearable inertial sensors for human motion analysis: a review. IEEE Sens J 16(22):7821–7834

    Article  Google Scholar 

  34. Wang Z, Guo M, Zhao C (2016) Badminton stroke recognition based on body sensor networks. IEEE Trans Human-Mach Syst 46(5):769–775

    Article  Google Scholar 

  35. Lau H, Tong K (2008) The reliability of using accelerometer and gyroscope for gait event identification on persons with dropped foot. Gait Posture 27(2):248–257

    Article  Google Scholar 

  36. Skog I, Handel P, Nilsson JO, Rantakokko J (2010) Zero-velocity detection-An algorithm evaluation. IEEE Trans Biomed Eng 57(11):2657–2666

    Article  Google Scholar 

  37. Hashmi MA, Riaz Q, Zeeshan M, Shahzad M, Fraz MM (2020) Motion reveal emotions: identifying emotions from human walk using chest mounted smartphone. IEEE Sens J 20(22):13511–13522

    Article  Google Scholar 

  38. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  39. Jeon H, Kim SL, Kim S, Lee D (2020) Fast wearable sensor? Based foot-ground contact phase classification using a convolutional neural network with sliding-window label overlapping. Sensors 20(17):4996

    Article  Google Scholar 

  40. Zhang H, Xue J, Dana K (2017) Deep ten: texture encoding network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 708–717

  41. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  42. Liu X, Chen M, Liang T, Lou C, Wang H, Liu X (2022) A lightweight double-channel depthwise separable convolutional neural network for multimodal fusion gait recognition. Math Biosci Eng 19:1195–1212

    Article  MATH  Google Scholar 

  43. Huang H, Zhou P, Li Y, Sun F (2021) A lightweight attention-based CNN model for efficient gait recognition with wearable IMU sensors. Sensors 21(8):2866

    Article  Google Scholar 

  44. Middya AI, Roy S, Mandal S, Talukdar R (2021) Privacy protected user identification using deep learning for smartphone-based participatory sensing applications. Neural Comput Appl 33:17303–17313

    Article  Google Scholar 

  45. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Houlsby N (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China under (Grant Nos. 61903170, 62173175, 61877033), and by the Natural Science Foundation of Shandong Province under grants Nos. ZR2019BF045, ZR2019MF021, ZR2019QF004, and by the Key Research and Development Project of Shandong Province of China, No. 2019GGX101003.

Author information

Authors and Affiliations

  1. School of Automation and Electrical Engineering, Linyi University, Lanshan Street, Linyi, 276000, Shandong Province, China

    Mengxue Yan, Ming Guo, Jianqiang Sun, Jianlong Qiu & Xiangyong Chen

Authors
  1. Mengxue Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianqiang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianlong Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiangyong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ming Guo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, M., Guo, M., Sun, J. et al. Gait Recognition in Different Terrains with IMUs Based on Attention Mechanism Feature Fusion Method. Neural Process Lett 55, 10215–10234 (2023). https://doi.org/10.1007/s11063-023-11324-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11324-y

Keywords

Search

Navigation