GaitGANv2: Invariant gait feature extraction using generative adversarial networks
Introduction
Gait is a behavioural biometric modality with a great potential for person identification because of its unique advantages such as being contactless, hard to fake and passive in nature, i.e., it requires no explicit cooperation from the subjects. Furthermore, the gait features can be captured at a distance in uncontrolled scenarios. Therefore, gait recognition is a very valuable technique in video surveillance, with a wide-ranging applications. This is particular so since many surveillance cameras have already been installed in major cities around world. Therefore, by continually improving its accuracy, the gait recognition technology will certainly add to the repertoire of tools available for crime prevention and forensic identification. For this reason, gait recognition is and will become an ever more important research topic in the computer vision community.
Unfortunately, automatic gait recognition remains a challenging task because it suffers from many potential sources of variation that can alter the human appearance drastically, such as, but not limited to aspects such as viewpoint, clothing, and objects being carried. These variations can affect the recognition accuracy greatly. Among these sources of variation, view angle is one of the most common one because we can not control the walking directions of subjects in real applications, and that is the central focus of our work here.
As a proof of concept, we shall consider variability in conditions of consisting of view angle, choice of clothing and type of objects being carried by the subject. The proposed generative adversarial networks (GAN) can handle all these variations simultaneously by using only one model. GAN acts as a regressor which takes an gait image captured with any combination of the above-mentioned sources of variation and then transforms it into a canonical side view image The method can do so without any knowledge regarding the factors that contribute to the gait variability. The most important computational challenge, however, is to address how to retain useful identity information when generating the canonical, invariant gait images.
The rest of the paper is organized as follows. Section 2 presents the state-of-the-art literature in gait recognition that deals with invariance in gait recognition. Section 3 describes the proposed method. Experiments and evaluation are presented in Section 4. The last section, Section 5, gives the conclusions and identifies future work.
Section snippets
Related work
To reduce the effect of different kinds of variations is what is concerned about by most gait recognition methods. Early literature such as [1] uses static body parameters measured from gait images as a kind of view-invariant feature. Kale et al. [2] used the perspective projection model to generated side view features from arbitrary views. Unfortunately, the relation between two views is hard to be modelled by a simple linear function, which is achieved via the perspective projection model.
Proposed method
To reduce the effect of variations, we propose to use GAN as a regressor to generate an invariant canonical gait image. The generated canonical image contains a subject’s gait viewed from the side, wearing a normal (standardized) cloth but without carrying anything. Any gait image appearing from any arbitrary poses is converted to the above canonical view because it contains richer information about the gait dynamics. While this is intuitively appealing, a key challenge that must be addressed
Datasets
To evaluate the proposed method, two datasets are involved. One is CASIA-B with 124 subjects and another is OU-ISIR Large Population Dataset with 4007 subjects.
CASIA-B gait dataset [33] is one of the popular public gait datasets which has been widely used to evaluate different gait recognition methods. It was created by the Institute of Automation, Chinese Academy of Sciences in January 2005. It consists of 124 subjects (31 females and 93 males) captured from 11 views. The view range is from 0°
Conclusions and future work
In this paper, we applied GaitGANv2 which is a variant of generative adversarial networks, PixelDTGAN, adopted to deal with variations in viewpoint, clothing and carrying conditions simultaneously in gait recognition. Extensive experiments on two large datasets show that the GaitGANv2 can transform gait images obtained from any viewpoint to the side view and remove the variations of clothings and carrying without the need to estimate the subject’s view angle, clothing type and carrying
Acknowledgment
The authors would like to thank Dr. Xianglei Xing for his support on experimental results of some methods. The work is supported by the Science Foundation of Shenzhen (Grant No. JCYJ20150324141711699 and 20170504160426188).
Shiqi Yu received his B.E. degree in computer science and engineering from the Chu Kochen Honors College, Zhejiang University in 2002, and Ph.D. degree in pattern recognition and intelligent systems from the Institute of Automation, Chinese Academy of Sciences in 2007. He worked as an assistant professor and then as an associate professor in the Shenzhen Institutes of Advanced Technology, Chinese Academy of Science from 2007 to 2010. Currently, he is an associate professor in the College of
References (37)
- et al.
Complete canonical correlation analysis with application to multi-view gait recognition
Pattern Recognit.
(2016) - et al.
An improved biometrics technique based on metric learning approach
Neurocomputing
(2012) - et al.
On the distance metric learning between cross-domain gaits
Neurocomputing
(2016) - et al.
Invariant feature extraction for gait recognition using only one uniform model
Neurocomputing
(2017) - et al.
GaitGAN: invariant gait feature extraction using generative adversarial networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops
(2017) - et al.
A multi-view method for gait recognition using static body parameters
Proceedings of the 3rd International Conference on Audio and Video Based Biometric Person Authentication
(2001) - et al.
Towards a view invariant gait recognition algorithm
Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance
(2003) - et al.
3d gait recognition using multiple cameras
Proceedings of the International Conference on Automatic Face and Gesture Recognition
(2006) - et al.
Model-based 3d gait biometrics
Proceedings of the International Joint Conference on Biometrics
(2011) - et al.
Robust arbitrary-view gait recognition based on 3d partial similarity matching
IEEE Trans. Image Process.
(2017)
Gait recognition using a view transformation model in the frequency domain
Proceedings of the ECCV
Multiple views gait recognition using view transformation model based on optimized gait energy image
Proceedings of the ICCV Workshops
Robust view transformation model for gait recognition
Proceedings of the ICIP
Gait recognition under various viewing angles based on correlated motion regression
IEEE TCSVT
Cross-view gait recognition using correlation strength
Proceedings of the BMVC
Robust gait recognition based on partitioning and canonical correlation analysis
Proceedings of the IEEE International Conference on Imaging Systems and Techniques
Human identity and gender recognition from gait sequences with arbitrary walking directions
IEEE TIFS
View-invariant discriminative projection for multi-view gait-based human identification
IEEE TIFS
Cited by (81)
A Faster R-CNN and recurrent neural network based approach of gait recognition with and without carried objects
2022, Expert Systems with ApplicationsCitation Excerpt :It cannot deal with the situation if a person carries various types of COs like backpack, drawstring bag, satchel backpack, shoulder bag, waist bag, handheld bag, suitcase, briefcase, trolley bag, etc. Similarly, the GAN based approaches (Yu et al., 2019) successfully remove the COs from the input images but these approaches regenerate other non-CO parts (for example, leg portions of the person carrying a drawstring bag) as well that may lead to unnecessary errors. So, the present investigation has considered the walking patterns both with and without COs and this article proposes a novel method using Faster R-CNN to detect and extract the pedestrian without CO (in case the pedestrian is carrying any CO) from the input image.
PoseMapGait: A model-based gait recognition method with pose estimation maps and graph convolutional networks
2022, NeurocomputingCitation Excerpt :To show the effectiveness of the pose estimation maps feature, we make comparisons with some advanced methods on the MoBo dataset. Including recent popular model-based method PoseGait [9], and appearance-based method GaitGANv2 [18], DV-GEIs-pre [4] and DV-GEIs [3]. We implemented these methods by ourselves as they do not cite the experimental results of the MoBo dataset from the original paper.
Deep learning gait recognition based on two branch spatiotemporal gait feature fusion
2024, Kongzhi yu Juece/Control and DecisionFrontal-view gait recognition using discriminative dynamics feature representations and learning
2024, Journal of Electronic Imaging
Shiqi Yu received his B.E. degree in computer science and engineering from the Chu Kochen Honors College, Zhejiang University in 2002, and Ph.D. degree in pattern recognition and intelligent systems from the Institute of Automation, Chinese Academy of Sciences in 2007. He worked as an assistant professor and then as an associate professor in the Shenzhen Institutes of Advanced Technology, Chinese Academy of Science from 2007 to 2010. Currently, he is an associate professor in the College of Computer Science and Software Engineering, Shenzhen University, China. He especially focuses on image classification and related research topics.
Rijun Liao received his B.S. degree from the College of Physics and Energy, Shenzhen University, China in 2015. He is currently a master student in the College of Computer Science and Software Engineering, Shenzhen University, China. His research interests include biometrics, computer vision and deep learning.
Weizhi An received her B.S. degree from the College of Computer Science and Software Engineering, Shenzhen University, China in 2016. She is currently a master student in the College of Computer Science and Software Engineering, Shenzhen University, China. Her research interests include biometrics, computer vision and deep learning.
Haifeng Chen received his B.S. degree in computer science and engineering from Qufu Normal University, China, in 2013, and his master degree from the College of Computer Science and Software Engineering, Shenzhen University, China, in 2017. His research interestsinclude computer vision and deep learning.
Edel B. Garcia Reyes is graduated of Mathematic and Cybernetic from University of Havana, in 1986 and received the Ph.D. in Technical Sciences at the Technical Military Institute ”Jose Marti” of Havana, in 1997. At the moment, he is working as a researcher in the Advanced Technologies Application Center. Dr. Edel has focused his researches on digital image processing of remote sensing data, biometrics and video surveillance. He has participated as member of technical committees and experts groups and has been reviewer for different events and journals as Pattern Recognition Letter, Journal of Real-Time Image Processing, etc. Dr. Edel worked in the Cuban Institute of Geodesy and Cartography (1986-1995) and in the Enterprise Group GeoCuba (1995-2001) where he was the head of the Agency of the Centre of Data and Computer Science of Geocuba - investigation and Consultancy (1998-2001).
Yongzhen Huang received the B.E. degree from the Huazhong University of Science and Technology in 2006 and the Ph.D. degree from the Institute of Automation, Chinese Academy of Sciences (CASIA) in 2011. In July 2011, he joined the National Laboratory of Pattern Recognition (NLPR), CASIA, where he is currently an associate professor. He has published more than 50 papers in the areas of computer vision and pattern recognition at international journals such as IEEE Transactions on Pattern Analysis and Machine Intelligence, International Journal of Computer Vision, IEEE Transactions on Systems, Man, and Cybernetics, IEEE Transactions on Circuits and Systems for Video Technology, IEEE Transactions on Multimedia, and conferences such as CVPR, ICCV, NIPS, and BMVC. His current research interests include pattern recognition, computer vision, and machine learning.
Norman Poh currently serves as CSO for Truststamp Europe and Data Scientist for BJSS London. He holds a PhD in Machine Learning and Information Fusion from IDIAP research institute, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland. He is passionate about machine learning with applications to biometric person recognition, healthcare, forensics, financial forecasting, and other practical data intensive areas, where he published more than 100 peer-reviewed publications, including 5 best paper awards. Previously, he was a Senior Lecturer at University of Surrey where he conducted research as principal investigator of two personal fellowship/grant schemes, i.e., Swiss NSF Advanced Researcher Award and Medical Research Council’s New Investigator Research Grant. He was named Researcher of the Year, University of Surrey in 2011.