Enhancing person re-identification by integrating gait biometric
Introduction
With the growing demands for security surveillance, large networks of cameras are deployed in public places such as college campuses and office buildings. These cameras have non-overlapping fields-of-views (FOVs) that provide huge amounts of video data. It is critical to understanding behavior of people in public spaces that a person must be re-identified in the FOV of one camera when he disappears from another one. Tracking these people by human monitoring is time consuming and expensive, thereby reducing the effectiveness of surveillance. Automated analysis of this inter-camera people association problem is known as person re-identification [1], which not only processes more efficiently but significantly improves the quality of surveillance.
The application scene of person re-identification is complicated that the monitored environment is uncontrolled [2]. Potentially, people may be caught by different cameras from different angles and distance, different background as well. Furthermore, the lighting conditions, degrees of occlusion maybe different and other view-specific variables may exist. The conventional biometrics of identification become unreliable because of the insufficiently constrained conditions. For instance, face recognition is noneffective if people are monitored from long distance and the resolution of videos is low. Besides, the geographic distribution of the cameras is hardly known. Thus, temporal and spatial constraint cannot be imposed accurately.
Designing suitable representation for person re-identification is a critical and challenging problem. Ideally, the selected features should be robust to the variation of illumination, posture and view. They should be able to handle background clutter, occlusion and low image quality as well. Some excellent appearance-based feature extraction methods have been proposed since appearance features are perceived as the most ideal way to describe people in person re-identification. Farenzena et al. [3] used color histograms, MSCR [4] as well as RHSP with the symmetry property to describe a local patch. Gray and Tao [5] let a machine learning algorithm find the best representation. Bazzani et al. [6] condensed a set of frames of an individual into a highly informative signature, called the Histogram Plus Epitome (HPE). Zhao et al. [7] found salience of people based on SIFT [8] and color histogram and then match patches with constraint.
However, appearance features have their own limitation. The main disadvantages of appearance features can be categorized as following.
- •
With the variation of view angles and illumination, appearance of people changes largely. Then, the intra-class variation can be large, even larger than the inter-class variation, namely, a person maybe more similar with another person in different views than the corresponding one.
- •
Color is distorted by different camera intrinsic parameters that cause the same color seems different in different cameras.
- •
In real world, people would probably to appear in public spaces in dark clothes in winter or wearing suit in official area. Thus, appearance based features are not informative about identity.
As above, biometrics are seldom mentioned in those proposed approaches. However, biometrics would play a significant role in person re-identification problem if used properly. In recent years, gait was proposed and gradually attracted attention of many scholars. There are many methods have been proposed about gait recognition. Little and Boyd [9] developed the shape of motion which is a model-free description of instantaneous motion, and used it to recognize individuals by their gait. Sarkar et al. [10] measured the similarity between the probe sequence and the gallery sequence directly by computing the correlation of corresponding frame pairs.
Gait feature is an ideal way for application in security surveillance as a biometric, because it can be obtained just by a camera from a long distance away. In addition, carrying temporal information is the other good quality of gait features. Instead of using single-shot images, image sequences grabbed from videos are used to generate temporal features. This process is closer to the real surveillance scenario considering inputs of cameras are sequences of images. Thus, a spatio-temporal analysis for person re-identification can be performed by integrating gait feature with appearance features.
For these reasons, gait can be used to enhance sequence based person re-identification. Gait is robust even if people change their appearance, such as in long-period person re-identification or crime scenes. In the contrary, appearance features are helpless in those situations. Gait is not such strong discriminative as traditional biometric. However, it is much more convenient to fuse gait with appearance features extracted from surveillance videos. From this perspective, comparing to recognition problem, gait is more suitable for person re-identification. Since a fusion step has to be processed, the selection of features and fusion strategies are important and hard work.
In this paper, we try to solve the person re-identification problem by integrating a gait feature with appearance features. For appearance features, HSV histogram is widely used to describe the color information, and the texture information can be well represented by Gabor feature [11]. Gait Energy image (GEI) [12] is a popular feature that used to represent gait biometric. Hence, the descriptor of the proposed method is composed of HSV histogram and Gabor feature as the appearance feature and GEI as the gait feature. Then these two types of features are fused by two strategies, namely score-level fusion and feature-level fusion. After the descriptive features are extracted, a metric learning method is adopted for similarity measurement and descriptors are matched by distances under the metric instead of Euclidean distance. Finally, we test our method on CASIA dataset which is a gait dataset created by Chinese Academy of Sciences (Fig. 1).
In summary, the main contributions of this work are three-fold as follows.
- •
For the first time, gait biometric is integrated into the descriptor for person re-identification, which has well performances whether people change their appearance or not.
- •
Two fusion strategies are proposed to fuse appearance based features and gait which well combine the advantages of appearance based features and gait and overcome the limitations of appearance based methods.
- •
The descriptors of people are matched with a metric learned by metric learning to make the proposed method more discriminative and effective.
Section snippets
Related works
Person re-identification has attracted increasing attentions in recent years. In most of the works, person re-identification is treated as a retrieval problem. Given an image or image sequence of an unknown person as probe, the works first build representative features to describe the person and then compute similarities between the probe person and all samples in a gallery set. Generally, a ranked list of all the people in gallery is built that the higher ranked person is more likely to have
Proposed framework
As an overview, the framework proposed in this paper contains hierarchical feature extraction, fusion strategy and descriptor matching. The appearance feature and gait feature are two parts of the descriptor. The appearance feature of people consists of HSV color histogram and Gabor filter. Gait Energy Image is generated by silhouette images to obtain the spatio-temporal information and Principal Component Analysis (PCA) [27] is used to obtain the low-dimension GEI feature. The GEI feature is
Fusion strategy
In order to effectively use the GEI feature and appearance features together, two fusion methods in the framework will be introduced in the following. The first one is the score-level fusion that fuse distances which are calculated by GEI feature and the appearance feature, respectively. The second one is the feature-level fusion that installs two type features in series.
Dataset
The proposed method is tested on the CASIA Gait Database B created by The Institute of Automation, Chinese Academy of Sciences (CASIA). This dataset contains eleven views of 124 individuals with three conditions. These three conditions are ‘bag’, which means the pedestrians appear with a bag in the video, ‘clothes’, which means the pedestrians appear with coat in the video and ‘normal’ means pedestrians appear without any coat or bag. The view angle contains 0°, 18°, 36°, 54°, 72°, 90°, 108°,
Conclusions
We attempt to enhance person re-identification by integrating gait biometric. In particular, we propose a hierarchical feature extraction method which extract appearance features and gait feature. The feature-level fusion and score-level fusion are adopted as strategies to fuse two types of features. A matching method based on the metric learned by MLR is also introduced to effectively re-identity people based on the hierarchical feature. Our experiments indicate that appearance features are
Acknowledgment
This work is funded by the National Natural Science Foundation of China (No. 61375036), the Beijing Natural Science Foundation (No. 4132064), the Program for New Century Excellent Talents in University, the Beijing Higher Education Young Elite Teacher Project, and the Fundamental Research Funds for the Central Universities.
Zheng Liu received his B.Eng. degree in computer science and technology from Beihang University, China (BUAA), in 2013, and is a Ph.D. candidate in the Laboratory of Intelligent Recognition and Image Processing (IRIP), School of Computer Science and Engineering, Beihang University, China. His research interests include computer vision, pattern recognition, and machine learning.
References (36)
- et al.
Multiple-shot person re-identification by chromatic and epitomic analyses
Pattern Recognit. Lett.
(2012) - et al.
A survey of approaches and trends in person re-identification
Image Vis. Comput.
(2014) Two-dimensional spectral analysis of cortical receptive field profiles
Vis. Res.
(1980)- Riccardo Satta. Appearance descriptors for person re-identification: a comprehensive review, arXiv preprint, 2013....
- et al.
Reidentification by relative distance comparison
IEEE Trans. Pattern Anal. Mach. Intell.
(2013) - Michela Farenzena, Loris Bazzani, Alessandro Perina, Vittorio Murino, Marco Cristani, Person re-identification by...
- P.-E. Forssén. Maximally stable colour regions for recognition and matching, in: IEEE Conference on Computer Vision and...
- Douglas Gray, Hai Tao. Viewpoint invariant pedestrian recognition with an ensemble of localized features, in: Computer...
- Rui Zhao, Wanli Ouyang, Xiaogang Wang, Unsupervised salience learning for person re-identification, in: 2013 IEEE...
Distinctive image features from scale-invariant keypoints
Int. J. Comput. Vis.
(2004)
Recognizing people by their gaitthe shape of motion
Videre: J. Comput. Vis. Res.
The humanid gait challenge problemdata sets, performance, and analysis
IEEE Trans. Pattern Anal. Mach. Intell.
Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition
IEEE Trans. Image Process.
Individual recognition using gait energy image
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (63)
Multi-view gait recognition system using spatio-temporal features and deep learning
2021, Expert Systems with ApplicationsCitation Excerpt :By introducing stereo images or small overlapping frames, the performance of the network is boosted. Zheng et al. (Liu, Zhang, Wu, & Wang, 2015) attempted to enhance re-identification of humans and make it robust by fusion of information regarding appearance and gait. The feature descriptor comprises of HSV histogram and Gabor feature for appearance and GEI as a gait biometric.
Privacy disclosure by de-anonymization using music preferences and selections
2021, Telematics and InformaticsA review of video surveillance systems
2021, Journal of Visual Communication and Image RepresentationCitation Excerpt :The similarity measures can be the key to computing the matching between two or a set of images. However, the re-identification using video clips can be a problem for many applications [158]. For example, people tracking across multiple cameras.
BiMPeR: A novel Bi-Model Person Re-identification method based on the appearance and the gait features
2021, Procedia Computer ScienceLearning 3D spatiotemporal gait feature by convolutional network for person identification
2020, NeurocomputingCitation Excerpt :It can be realized that some biometric identifiers ask for a direct interaction between the subject and data acquisition devices (e.g., a fingerprint is analyzed for individual identification through an action of touching the optical sensor) while the others can automatically execute from a distance. In popular surveillance systems, the identification process needs to be done without subject notice [3], therefore, visual approaches like face and gait recognition are regularly considered for the deployment in public regions (e.g., shopping mall and subway station) [4]. Especially, gait-based person identification also possesses several advantages of non-subject-cooperation and robustness against cyberattacks in comparison with other biometric identification technologies [5].
Zheng Liu received his B.Eng. degree in computer science and technology from Beihang University, China (BUAA), in 2013, and is a Ph.D. candidate in the Laboratory of Intelligent Recognition and Image Processing (IRIP), School of Computer Science and Engineering, Beihang University, China. His research interests include computer vision, pattern recognition, and machine learning.
Zhaoxiang Zhang received the B.S. degree in electronic science and technology from the University of Science and Technology of China, Hefei, China, in 2004, and the Ph.D. degree in pattern recognition and intelligent systems from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 2009. He joined the Laboratory of Intelligent Recognition and Image Processing, Beijing Key Laboratory of Digital Media, School of Computer Science and Engineering, Beihang University, Beijing, as a Faculty Member, in 2009. His research interests include computer vision, pattern recognition, image processing, and machine learning.
Qiang Wu received the B.Eng. and M.Eng. degrees in electronic engineering from Harbin Institute of Technology, China, in 1996 and 1998, and the Ph.D. degree in computing science from University of Technology, Sydney, Australia, in 2004. In 2003, he joined in the School of Computing and Communications, University of Technology, Sydney (UTS), Australia, where he is currently a associate professor. His research interests include computer vision, image processing, pattern recognition, machine learning, and multimedia processing. He is a member of the IEEE.
Yunhong Wang received the B.S. degree in electronic engineering from Northwestern Polytechnical University, Xi’an, China, in 1989 and the M.S. and Ph.D. degrees in electronic engineering from Nanjing University of Science and Technology, Nanjing, China, in 1995 and 1998, respectively. From 1998 to 2004, she was with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China. Since 2004, she has been a Professor with the School of Computer Science and Engineering, Beihang University, Beijing, where she is also the Director of the Laboratory of Intelligent Recognition and Image Processing, Beijing Key Laboratory of Digital Media. Her research interests include biometrics, pattern recognition, computer vision, data fusion, and image processing. Dr. Wang is a member of the IEEE Computer Society.