Fusion of spatial-temporal and kinematic features for gait recognition with deterministic learning
Introduction
Since September 11th attack, the demand for automatic human identification is strongly increasing and growing, especially noncontact human identification at a distance. In security-sensitive environments (e.g. railway stations, airports and banks), it is desirable to detect threats quickly and biometrics is a suitable, powerful tool for reliable human identification [1].
As a new behavioral biometric, gait recognition aims at identifying people by the way they walk. Compared with other widely used biometrics, the main characteristics of gait recognition lie in the following aspects:
- 1.
Gait is unique. From a biomechanics perspective, gait is unique for each person if all the properties of body structures, synchronized integrated movements of body parts, interaction among them are considered. The potential of gait for automatic human identification is supported by a rich literature [2].
- 2.
Gait is noncontact. The first generational biometrics, such as face, fingerprints and iris, are restricted to controlled environments, usually require physical touch or proximal sensing. In contrast, gait has great prominent advantages of being non-contact, non-invasive, unobvious. Gait can be collected secretly, which does not require the subject cooperation [3].
- 3.
Gait can be collected at a distance. Biometrics such as fingerprint and iris usually require sensing the subject at close ranges. However, at a distance, these biometrics are no longer applicable. Fortunately, gait can still work in this case, even in a low resolution environment. This makes gait ideal for long distance security and surveillance applications [4].
As stated above, gait has many advantages, making it very attractive for human identification at a distance and applications in video surveillance.
Existing gait recognition methods mainly fall into two categories: model-based methods and silhouette-based methods [5].
Model-based methods model the human body and its motion from gait sequences. Kinematic characteristics of walking are then extracted from the model components and used as features for classification. Cunado et al. [6] proposed an early gait-pendulum model and achieved model-based gait recognition. Nixon et al. [7] developed a stick model and calculated walking kinematic characteristics without directly analyzing gait sequences. Mu and Wu [8] presented a five-link bipedal walking model. More recently, techniques based on activity-specific static body parameters [9], deterministic learning and five-link model were developed for model-based gait recognition [10].
Silhouette-based methods directly operate on the gait sequences without any specific model. Gait characteristics are implicitly reflected by the holistic appearance of walking individual. Phillips et al. [11] used the silhouettes features to establish a baseline recognition algorithm. Han et al. [12] characterized human gait pattern with Gait Energy Image (GEI) by averaging image of silhouettes in one gait period. Alpha-GEI, an enhanced version of GEI, was proposed by Hofmann et al. [13] to mitigate nonrandom noise. Makihara et al. [14] extracted individuality-preserving silhouette for gait recognition. Matovski et al. [15] improved the segmentation processing by using quality metrics for automatic gait recognition. More recently, techniques based on gait entropy image (GEnI) [16], chrono gait image (CGI) [17] were developed for silhouette-based gait recognition.
The most commonly used gait features, according to human gait theory, can be roughly divided into two categories: spatial-temporal parameters and kinematic parameters [18]. Generally speaking, spatial-temporal parameters are the intuitive gait features including stride length, step length, silhouette width and so on. Kinematic parameters are usually characterized by the joint angles between body segments and joint motion in the gait cycle [18]. In [19], step length and speed were extracted as spatial-temporal parameters to perform the gait recognition task. In [20], lower limb angles were extracted as kinematic parameters. In [21], step length, cycle time, speed and angle-based kinematic parameters were combined as gait features. Vertical distance features (VDF) was developed by Ahmed et al. [22]. Chattopadhyay et al. [23] attempted to combine the relative distance features and joint velocity to achieve better performance.
In our previous works [10], [24], the potential of the use of the spatial-temporal and the kinematic parameters in gait recognition has been investigated separately. In [10], the dynamics along the phase portrait of joint angles versus angular velocities were captured to achieve model-based gait recognition. In [24], the dynamics along the trajectories of silhouette width features were captured to achieve silhouette-base gait recognition. The experimental results indicated that, for the purpose of gait recognition, the amount of discriminability provided by the dynamics of the silhouette feature is similar (or equivalent) to the discriminability provided by the dynamics of kinematic parameters like joint angles and/or angular velocities [24]. However, the combined use of silhouette spatial-temporal feature and kinematic parameters has not been investigated in our experiments yet.
For obtaining optimal performance, as many informative cues as possible should be involved in the gait recognition algorithm. Based on this assumption, in this paper, we attempt to fuse the two completely different sources of information: spatial-temporal and the kinematic parameters for human identification.
The proposed method is schematically shown in Fig. 1. For each gait sequence, lower limbs silhouette widths and holistic silhouette area are extracted as spatial-temporal parameters, lower limbs joint angles are extracted as kinematic parameters. Spatial-temporal gait features can then be calculated using deterministic learning algorithm and be represented as the dynamics along the trajectories of lower limbs silhouette widths and holistic silhouette area. Additionally, kinematic gait features can be extracted and represented as the dynamics along the trajectories of four lower limbs joint angles. This two kinds of gait features reflect the temporal change of body poses or walking motion between consecutive frame sequences in two completely different aspects, while preserving temporal dynamics information of human walking. Both spatial-temporal and kinematic information can be independently used for recognition using the smallest error principle. They are combined as well on the decision level for a better recognition performance.
Section snippets
Spatial-temporal feature extraction
As schematically shown in Fig. 2, spatial-temporal parameters from each gait sequence are extracted, spatial-temporal gait features can then be calculated using deterministic learning algorithm and be represented as the dynamics along the trajectories of spatial-temporal parameters. In deterministic learning algorithm, identification of nonlinear gait system dynamics is achieved according to the following elements: (a) employment of localized radial basis function (RBF) networks; (b)
Kinematic feature extraction
As schematically shown in Fig. 6, a five-link biped model is selected and the dynamics of the biped model is derived. With the absolute domination of gait dynamics, four lower limbs joints angles are extracted and selected as kinematic parameters. Kinematic gait features can then be calculated using deterministic learning algorithm.
Recognition scheme and fusion rules
As a traditional pattern recognition problem, gait recognition in this paper can be achieved by measuring similarities between training gait signature matrixs and test signature matrixs . Here we try the smallest error principle. The following summarizes the main steps in recognizing a test gait sequence using the smallest error principle.
First, a bank of M estimators is constructed for the trained sequences by using the learned knowledge obtained in the training phase:
Experiments
In this section, five widely used gait databases: (1) CASIA gait database B; (2) CASIA gait database C; (3) TUM GAID gait database; (4) OU-ISIR treadmill gait database B; (5) USF HumanID database are used to evaluate the performance of the proposed method.
Conclusion
The fusion of spatial-temporal and kinematic features is investigated in this paper for human gait recognition. There are some conclusions in below.
Deterministic learning theory is used to extract the gait dynamics underlying spatial-temporal and kinematic parameters. Spatial-temporal gait features can be represented as the gait dynamics underlying the trajectories of spatial-temporal parameters, which can implicitly reflect the temporal changes of silhouette shape. Kinematic gait features can
Acknowledgments
This work was supported by the National Science Fund for Distinguished Young Scholars (Grant No. 61225014), by the National RD Program for Major Research Instruments (Grant No. 61527811).
Muqing Deng is a Ph.D. candidate at the College of Automation, South China University of Technology, Guangzhou, China. His current research interests include dynamical pattern recognition, gait recognition, deterministic learning theory.
References (58)
- et al.
Recent developments in human motion analysis
Pattern Recognit.
(2003) - et al.
Human gait recognition via deterministic learning
Neural Networks
(2012) - et al.
The humanid gait challenge problem: data sets, performance, and analysis
IEEE Trans. Pattern Anal. Mach. Intell.
(2005) - et al.
Deterministic learning and rapid dynamical pattern recognition
IEEE Trans. Neural Networks
(2007) - et al.
Automatic gait recognition using weighted binary pattern on video
Advanced Video and Signal Based Surveillance, 2009. AVSS’09. Sixth IEEE International Conference on
(2009) - et al.
Dynamic distance-based shape features for gait recognition
J Math Imaging Vis
(2014) - et al.
Gait probability image: an information-theoretic model of gait representation
J. Vision Commun. Image Represent.
(2014) - et al.
Gait flow image: a silhouette-based gait representation for human identification
Pattern Recognit.
(2011) - et al.
Gait recognition based on shape and motion analysis of silhouette contours
Comput. Vision Image Understanding
(2013) - et al.
Eigengait: motion-based recognition of people using image self-similarity
International Conference on Audio-and Video-Based Biometric Person Authentication
(2001)
Low-resolution gait recognition
IEEE Trans. Syst. Man. Cybernet. B
Fusion of static and dynamic body biometrics for gait recognition
IEEE Trans. Circuits Syst. Video Technol.
Gait recognition
Computer Vision
Using gait as a biometric, via phase-weighted magnitude spectra
International Conference on Audio-and Video-Based Biometric Person Authentication
Extracting gait signatures based on anatomical knowledge
Proceedings of BMVA Symposium on Advancing Biometric Technologies
A complete dynamic model of five-link bipedal walking
Proceedings of the 2003 American Control Conference
Gait recognition using static, activity-specific parameters
International Conference on Computer Vision and Pattern Recognition
The gait identification challenge problem: data sets and baseline algorithm
2002 Proceedings. 16th International Conference on Pattern Recognition
Individual recognition using gait energy image
IEEE Trans. Pattern Anal. Mach. Intell.
The gait identification challenge problem: data sets and baseline algorithm
2012 5th IAPR International Conference on Biometrics (ICB)
Individuality-preserving silhouette extraction for gait recognition
IPSJ Trans. Comput. Vision Appl.
On including quality in applied automatic gait recognition
2012 21st International Conference on Pattern Recognition (ICPR)
Gait recognition using gait entropy image
3rd International Conference on Crime Detection and Prevention (ICDP 2009)
Chrono-gait image: a novel temporal template for gait recognition
European Conference on Computer Vision
Automated markerless analysis of human gait motion for recognition and classification
Etri J.
Gait recognition with kinect
1st International Workshop on Kinect in Pervasive Computing
Unsupervised clustering of people from’skeleton’data
Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction
Person identification using anthropometric and gait data from kinect sensor.
AAAI
Gait recognition based on kinect sensor
SPIE Photonics Europe
Cited by (66)
Deterministic learning-based neural identification and knowledge fusion
2024, Neural NetworksA Faster R-CNN and recurrent neural network based approach of gait recognition with and without carried objects
2022, Expert Systems with ApplicationsCitation Excerpt :These features were then embedded by the null Foley–Sammon transform (NFST). Deng, Wang, Cheng, and Zeng (2017) presented a gait recognition system where spatial–temporal and kinematic features were fused. The system has relied on deterministic learning.
Frame-level refinement networks for skeleton-based gait recognition
2022, Computer Vision and Image UnderstandingHuman identification system using 3D skeleton-based gait features and LSTM model
2022, Journal of Visual Communication and Image RepresentationCitation Excerpt :However, silhouette quality, i.e., segmentation errors, affect the performance of GEI [10]. Deng et al. [11] proposed a gait recognition algorithm based on the combination of spatial–temporal and kinematic gait features extracted from silhouette images. Model-based approaches analyze the shape and kinematics of human body parts for gait recognition.
Vision-based approaches towards person identification using gait
2021, Computer Science Review
Muqing Deng is a Ph.D. candidate at the College of Automation, South China University of Technology, Guangzhou, China. His current research interests include dynamical pattern recognition, gait recognition, deterministic learning theory.
Cong Wang received the B.E. and M.E. degrees from Beijing University of Aeronautics and Astronautics, Beijing, China, in 1989 and 1997, respectively, and the Ph.D. degree from the National University of Singapore, Singapore, in 2002. Currently, he is a professor at the College of Automation Science and Engineering, South China University of Technology, Guangzhou, China. He has authored and co-authored over 60 papers in international journals and conferences, and is a co-author of the book Deterministic Learning Theory for Identification, Recognition and Control (Boca Raton, FL: CRC Press, 2009). His current research interests include dynamical pattern recognition, adaptive NN control/identification, deterministic learning theory, pattern-based intelligent control, oscillation fault diagnosis, and cognitive and brain sciences.
Fengjiang Cheng is a M.S. candidate at the College of Automation and Center for Control and Optimization, South China University of Technology, Guangzhou, China. His current research interests include gait recognition, engineering application of deterministic learning theory.
Wei Zeng received the M.E. degree from the Department of Automation, Xiamen University, Xiamen, China, in 2008, and the Ph.D. degree from the College of Automation Science and Engineering, South China University of Technology, Guangzhou, China, in 2012. His current research interests include dynamical pattern recognition, adaptive NN control/identification, and deterministic learning theory.