Elsevier

Pattern Recognition

Volume 67, July 2017, Pages 186-200
Pattern Recognition

Fusion of spatial-temporal and kinematic features for gait recognition with deterministic learning

https://doi.org/10.1016/j.patcog.2017.02.014Get rights and content

Highlights

  • We present a gait recognition method based on the fusion of different features.

  • Spatial-temporal and kinematic features can fused for human identification.

  • We show good recognition performance on four widely used gait databases.

Abstract

For obtaining optimal performance, as many informative cues as possible should be involved in the gait recognition algorithm. This paper describes a gait recognition algorithm by combining spatial-temporal and kinematic gait features. For each walking sequence, the binary silhouettes are characterized with four time-varying spatial-temporal parameters, including three lower limbs silhouette widths and one holistic silhouette area. Using deterministic learning algorithm, spatial-temporal gait features can be represented as the gait dynamics underlying the trajectories of lower limbs silhouette widths and holistic silhouette area, which can implicitly reflect the temporal changes of silhouette shape. In addition, a model-based method is proposed to extract joint-angle trajectories of lower limbs. Kinematic gait features can be represented as the gait dynamics underlying the trajectories of joint angles, which can represent the temporal changes of body structure and dynamics. Both spatial-temporal and kinematic cues can be used separately for gait recognition using smallest error principle. They are fused on the decision level using different combination rules to improve the gait recognition performance. The fusion of two different kinds of features provides a comprehensive characterization of gait dynamics, which is not sensitive to the walking conditions variation. The proposed method can still achieve superior performance when the testing walking conditions are different from the corresponding training conditions. Experimental results show that encouraging recognition accuracy can be achieved on five public gait databases: CASIA-B, CASIA-C, TUM GAID, OU-ISIR, USF HumanID.

Introduction

Since September 11th attack, the demand for automatic human identification is strongly increasing and growing, especially noncontact human identification at a distance. In security-sensitive environments (e.g. railway stations, airports and banks), it is desirable to detect threats quickly and biometrics is a suitable, powerful tool for reliable human identification [1].

As a new behavioral biometric, gait recognition aims at identifying people by the way they walk. Compared with other widely used biometrics, the main characteristics of gait recognition lie in the following aspects:

  • 1.

    Gait is unique. From a biomechanics perspective, gait is unique for each person if all the properties of body structures, synchronized integrated movements of body parts, interaction among them are considered. The potential of gait for automatic human identification is supported by a rich literature [2].

  • 2.

    Gait is noncontact. The first generational biometrics, such as face, fingerprints and iris, are restricted to controlled environments, usually require physical touch or proximal sensing. In contrast, gait has great prominent advantages of being non-contact, non-invasive, unobvious. Gait can be collected secretly, which does not require the subject cooperation [3].

  • 3.

    Gait can be collected at a distance. Biometrics such as fingerprint and iris usually require sensing the subject at close ranges. However, at a distance, these biometrics are no longer applicable. Fortunately, gait can still work in this case, even in a low resolution environment. This makes gait ideal for long distance security and surveillance applications [4].

As stated above, gait has many advantages, making it very attractive for human identification at a distance and applications in video surveillance.

Existing gait recognition methods mainly fall into two categories: model-based methods and silhouette-based methods [5].

Model-based methods model the human body and its motion from gait sequences. Kinematic characteristics of walking are then extracted from the model components and used as features for classification. Cunado et al. [6] proposed an early gait-pendulum model and achieved model-based gait recognition. Nixon et al. [7] developed a stick model and calculated walking kinematic characteristics without directly analyzing gait sequences. Mu and Wu [8] presented a five-link bipedal walking model. More recently, techniques based on activity-specific static body parameters [9], deterministic learning and five-link model were developed for model-based gait recognition [10].

Silhouette-based methods directly operate on the gait sequences without any specific model. Gait characteristics are implicitly reflected by the holistic appearance of walking individual. Phillips et al. [11] used the silhouettes features to establish a baseline recognition algorithm. Han et al. [12] characterized human gait pattern with Gait Energy Image (GEI) by averaging image of silhouettes in one gait period. Alpha-GEI, an enhanced version of GEI, was proposed by Hofmann et al. [13] to mitigate nonrandom noise. Makihara et al. [14] extracted individuality-preserving silhouette for gait recognition. Matovski et al. [15] improved the segmentation processing by using quality metrics for automatic gait recognition. More recently, techniques based on gait entropy image (GEnI) [16], chrono gait image (CGI) [17] were developed for silhouette-based gait recognition.

The most commonly used gait features, according to human gait theory, can be roughly divided into two categories: spatial-temporal parameters and kinematic parameters [18]. Generally speaking, spatial-temporal parameters are the intuitive gait features including stride length, step length, silhouette width and so on. Kinematic parameters are usually characterized by the joint angles between body segments and joint motion in the gait cycle [18]. In [19], step length and speed were extracted as spatial-temporal parameters to perform the gait recognition task. In [20], lower limb angles were extracted as kinematic parameters. In [21], step length, cycle time, speed and angle-based kinematic parameters were combined as gait features. Vertical distance features (VDF) was developed by Ahmed et al. [22]. Chattopadhyay et al. [23] attempted to combine the relative distance features and joint velocity to achieve better performance.

In our previous works [10], [24], the potential of the use of the spatial-temporal and the kinematic parameters in gait recognition has been investigated separately. In [10], the dynamics along the phase portrait of joint angles versus angular velocities were captured to achieve model-based gait recognition. In [24], the dynamics along the trajectories of silhouette width features were captured to achieve silhouette-base gait recognition. The experimental results indicated that, for the purpose of gait recognition, the amount of discriminability provided by the dynamics of the silhouette feature is similar (or equivalent) to the discriminability provided by the dynamics of kinematic parameters like joint angles and/or angular velocities [24]. However, the combined use of silhouette spatial-temporal feature and kinematic parameters has not been investigated in our experiments yet.

For obtaining optimal performance, as many informative cues as possible should be involved in the gait recognition algorithm. Based on this assumption, in this paper, we attempt to fuse the two completely different sources of information: spatial-temporal and the kinematic parameters for human identification.

The proposed method is schematically shown in Fig. 1. For each gait sequence, lower limbs silhouette widths and holistic silhouette area are extracted as spatial-temporal parameters, lower limbs joint angles are extracted as kinematic parameters. Spatial-temporal gait features can then be calculated using deterministic learning algorithm and be represented as the dynamics along the trajectories of lower limbs silhouette widths and holistic silhouette area. Additionally, kinematic gait features can be extracted and represented as the dynamics along the trajectories of four lower limbs joint angles. This two kinds of gait features reflect the temporal change of body poses or walking motion between consecutive frame sequences in two completely different aspects, while preserving temporal dynamics information of human walking. Both spatial-temporal and kinematic information can be independently used for recognition using the smallest error principle. They are combined as well on the decision level for a better recognition performance.

Section snippets

Spatial-temporal feature extraction

As schematically shown in Fig. 2, spatial-temporal parameters from each gait sequence are extracted, spatial-temporal gait features can then be calculated using deterministic learning algorithm and be represented as the dynamics along the trajectories of spatial-temporal parameters. In deterministic learning algorithm, identification of nonlinear gait system dynamics is achieved according to the following elements: (a) employment of localized radial basis function (RBF) networks; (b)

Kinematic feature extraction

As schematically shown in Fig. 6, a five-link biped model is selected and the dynamics of the biped model is derived. With the absolute domination of gait dynamics, four lower limbs joints angles are extracted and selected as kinematic parameters. Kinematic gait features can then be calculated using deterministic learning algorithm.

Recognition scheme and fusion rules

As a traditional pattern recognition problem, gait recognition in this paper can be achieved by measuring similarities between training gait signature matrixs W¯training and test signature matrixs W¯test. Here we try the smallest error principle. The following summarizes the main steps in recognizing a test gait sequence using the smallest error principle.

First, a bank of M estimators is constructed for the trained sequences by using the learned knowledge obtained in the training phase: χ¯˙k=B(

Experiments

In this section, five widely used gait databases: (1) CASIA gait database B; (2) CASIA gait database C; (3) TUM GAID gait database; (4) OU-ISIR treadmill gait database B; (5) USF HumanID database are used to evaluate the performance of the proposed method.

Conclusion

The fusion of spatial-temporal and kinematic features is investigated in this paper for human gait recognition. There are some conclusions in below.

Deterministic learning theory is used to extract the gait dynamics underlying spatial-temporal and kinematic parameters. Spatial-temporal gait features can be represented as the gait dynamics underlying the trajectories of spatial-temporal parameters, which can implicitly reflect the temporal changes of silhouette shape. Kinematic gait features can

Acknowledgments

This work was supported by the National Science Fund for Distinguished Young Scholars (Grant No. 61225014), by the National R&D Program for Major Research Instruments (Grant No. 61527811).

Muqing Deng is a Ph.D. candidate at the College of Automation, South China University of Technology, Guangzhou, China. His current research interests include dynamical pattern recognition, gait recognition, deterministic learning theory.

References (58)

  • J. Zhang et al.

    Low-resolution gait recognition

    IEEE Trans. Syst. Man. Cybernet. B

    (2010)
  • L. Wang et al.

    Fusion of static and dynamic body biometrics for gait recognition

    IEEE Trans. Circuits Syst. Video Technol.

    (2004)
  • D.S. Matovski et al.

    Gait recognition

    Computer Vision

    (2014)
  • D. Cunado et al.

    Using gait as a biometric, via phase-weighted magnitude spectra

    International Conference on Audio-and Video-Based Biometric Person Authentication

    (1997)
  • J.-H. Yoo et al.

    Extracting gait signatures based on anatomical knowledge

    Proceedings of BMVA Symposium on Advancing Biometric Technologies

    (2002)
  • X. Mu et al.

    A complete dynamic model of five-link bipedal walking

    Proceedings of the 2003 American Control Conference

    (2003)
  • A.F. Bobick et al.

    Gait recognition using static, activity-specific parameters

    International Conference on Computer Vision and Pattern Recognition

    (2001)
  • P.J. Phillips et al.

    The gait identification challenge problem: data sets and baseline algorithm

    2002 Proceedings. 16th International Conference on Pattern Recognition

    (2002)
  • J. Man et al.

    Individual recognition using gait energy image

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2006)
  • M. Hofmann et al.

    The gait identification challenge problem: data sets and baseline algorithm

    2012 5th IAPR International Conference on Biometrics (ICB)

    (2012)
  • Y. Makihara et al.

    Individuality-preserving silhouette extraction for gait recognition

    IPSJ Trans. Comput. Vision Appl.

    (2015)
  • D.S. Matovski et al.

    On including quality in applied automatic gait recognition

    2012 21st International Conference on Pattern Recognition (ICPR)

    (2012)
  • K. Bashir et al.

    Gait recognition using gait entropy image

    3rd International Conference on Crime Detection and Prevention (ICDP 2009)

    (2009)
  • C. Wang et al.

    Chrono-gait image: a novel temporal template for gait recognition

    European Conference on Computer Vision

    (2010)
  • J.-H. Yoo et al.

    Automated markerless analysis of human gait motion for recognition and classification

    Etri J.

    (2011)
  • J. Preis et al.

    Gait recognition with kinect

    1st International Workshop on Kinect in Pervasive Computing

    (2012)
  • A. Ball et al.

    Unsupervised clustering of people from’skeleton’data

    Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction

    (2012)
  • V.O. Andersson et al.

    Person identification using anthropometric and gait data from kinect sensor.

    AAAI

    (2015)
  • M. Ahmed et al.

    Gait recognition based on kinect sensor

    SPIE Photonics Europe

    (2014)
  • Cited by (66)

    • A Faster R-CNN and recurrent neural network based approach of gait recognition with and without carried objects

      2022, Expert Systems with Applications
      Citation Excerpt :

      These features were then embedded by the null Foley–Sammon transform (NFST). Deng, Wang, Cheng, and Zeng (2017) presented a gait recognition system where spatial–temporal and kinematic features were fused. The system has relied on deterministic learning.

    • Frame-level refinement networks for skeleton-based gait recognition

      2022, Computer Vision and Image Understanding
    • Human identification system using 3D skeleton-based gait features and LSTM model

      2022, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      However, silhouette quality, i.e., segmentation errors, affect the performance of GEI [10]. Deng et al. [11] proposed a gait recognition algorithm based on the combination of spatial–temporal and kinematic gait features extracted from silhouette images. Model-based approaches analyze the shape and kinematics of human body parts for gait recognition.

    View all citing articles on Scopus

    Muqing Deng is a Ph.D. candidate at the College of Automation, South China University of Technology, Guangzhou, China. His current research interests include dynamical pattern recognition, gait recognition, deterministic learning theory.

    Cong Wang received the B.E. and M.E. degrees from Beijing University of Aeronautics and Astronautics, Beijing, China, in 1989 and 1997, respectively, and the Ph.D. degree from the National University of Singapore, Singapore, in 2002. Currently, he is a professor at the College of Automation Science and Engineering, South China University of Technology, Guangzhou, China. He has authored and co-authored over 60 papers in international journals and conferences, and is a co-author of the book Deterministic Learning Theory for Identification, Recognition and Control (Boca Raton, FL: CRC Press, 2009). His current research interests include dynamical pattern recognition, adaptive NN control/identification, deterministic learning theory, pattern-based intelligent control, oscillation fault diagnosis, and cognitive and brain sciences.

    Fengjiang Cheng is a M.S. candidate at the College of Automation and Center for Control and Optimization, South China University of Technology, Guangzhou, China. His current research interests include gait recognition, engineering application of deterministic learning theory.

    Wei Zeng received the M.E. degree from the Department of Automation, Xiamen University, Xiamen, China, in 2008, and the Ph.D. degree from the College of Automation Science and Engineering, South China University of Technology, Guangzhou, China, in 2012. His current research interests include dynamical pattern recognition, adaptive NN control/identification, and deterministic learning theory.

    View full text