research-article

Hybrid Modality Metric Learning for Visible-Infrared Person Re-Identification

Authors:

Jinqiao WangAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 18, Issue 1s

Article No.: 25, Pages 1 - 15

https://doi.org/10.1145/3473341

Published: 25 January 2022 Publication History

Abstract

Visible-infrared person re-identification (Re-ID) has received increasing research attention for its great practical value in night-time surveillance scenarios. Due to the large variations in person pose, viewpoint, and occlusion in the same modality, as well as the domain gap brought by heterogeneous modality, this hybrid modality person matching task is quite challenging. Different from the metric learning methods for visible person re-ID, which only pose similarity constraints on class level, an efficient metric learning approach for visible-infrared person Re-ID should take both the class-level and modality-level similarity constraints into full consideration to learn sufficiently discriminative and robust features. In this article, the hybrid modality is divided into two types, within modality and cross modality. We first fully explore the variations that hinder the ranking results of visible-infrared person re-ID and roughly summarize them into three types: within-modality variation, cross-modality modality-related variation, and cross-modality modality-unrelated variation. Then, we propose a comprehensive metric learning framework based on four kinds of paired-based similarity constraints to address all the variations within and cross modality. This framework focuses on both class-level and modality-level similarity relationships between person images. Furthermore, we demonstrate the compatibility of our framework with any paired-based loss functions by giving detailed implementation of combing it with triplet loss and contrastive loss separately. Finally, extensive experiments of our approach on SYSU-MM01 and RegDB demonstrate the effectiveness and superiority of our proposed metric learning framework for visible-infrared person Re-ID.

References

[1]

Yuanxin Zhu A, Zhao Yang A, Li Wang A, Sai Zhao A, Xiao Hu A, and Dapeng Tao B. 2020. Hetero-Center loss for cross-modality person Re-identification. Neurocomputing 386 (2020), 97–109.

[2]

Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. 2017. Beyond triplet loss: A deep quadruplet network for person re-identification. In Proceedings of the Conference on Computer Vision and Pattern Recognition.

[3]

De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. 2016. Person re-identification by multi-channel parts-based CNN with improved triplet loss function. In Computer Vision and Pattern Recognition.

[4]

Seokeon Choi, Sumin Lee, Youngeun Kim, Taekyung Kim, and Changick Kim. 2020. Hi-CMD: Hierarchical cross-modality disentanglement for visible-infrared person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE.

[5]

Pingyang Dai, Rongrong Ji, Haibin Wang, Qiong Wu, and Yuyu Huang. 2018. Cross-modality person re-identification with generative adversarial training. In Proceedings of the 27th International Joint Conference on Artificial Intelligence IJCAI-18.

Digital Library

[6]

N. Dalal. 2005. Histograms of oriented gradients for human detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’05).

Digital Library

[7]

Nguyen Dat, Hong Hyung, Kim Ki, and Park Kang. 2017. Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17, 3 (2017), 605. DOI:

[8]

Weijian Deng, Liang Zheng, Guoliang Kang, Yi Yang, Qixiang Ye, and Jianbin Jiao. 2018. Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE.

[9]

Zhangxiang Feng, Jianhuang Lai, and Xiaohua Xie. 2019. Learning modality-specific representations for visible-infrared person re-identification. IEEE Transactions on Image Processing 29 (2019), 579–590.

[10]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Y. Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS’14). 2672–2680.

Digital Library

[11]

Wang Guan’An, Zhang Tianzhu, Cheng Jian, Liu Si, Yang Yang, and Hou Zengguang. 2020. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment. In IEEE/CVF International Conference on Computer Vision (ICCV’19). IEEE.

[12]

Lup Hao, Jiang Wei, Fan Xing, and Zhang Sipeng. 2019. A survey on deep learning based person re-identification. Acta Automat. Sin. 45, 11 (2019), 2032–2049. DOI:

[13]

Yi Hao, Nannan Wang, Jie Li, and Xinbo Gao. 2019. HSME: Hypersphere manifold embedding for visible thermal person re-identification. Proc. AAAI Conf. Artif. Intell. 33 (2019), 8385–8392. DOI:https://doi.org/10.1609/aaai.v33i01.33018385

[14]

Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In defense of the triplet loss for person re-identification. CoRR abs/1703.07737 (2017). DOI:

[15]

Yan Huang, Jingsong Xu, Qiang Wu, Zhedong Zheng, Zhaoxiang Zhang, and Jian Zhang. 2018. Multi-pseudo regularized label for generated data in person re-identification. IEEE Transactions on Image Processing PP (2018), 1–1. https://doi.org/10.1109/TIP.2018.2874715

[16]

Mengxi Jia, Yunpeng Zhai, Shijian Lu, Siwei Ma, and Jian Zhang. 2020. A similarity inference metric for RGB-infrared cross-modality person re-identification. In Proceedings of the International Joint Conferences on Artificial Intelligence Organization.

Digital Library

[17]

Diangang Li, Xing Wei, Xiaopeng Hong, and Yihong Gong. 2020. Infrared-visible cross-modal person re-identification with an X modality. Proc. AAAI Conf. Artif. Intell. 34, 4 (2020), 4610–4617.

[18]

Shengcai Liao, Yang Hu, Xiangyu Zhu, and Stan Z. Li. 2015. Person re-identification by Local Maximal Occurrence representation and metric learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE.

[19]

Haijun Liu, Jian Cheng, Wen Wang, Yanzhou Su, and Haiwei Bai. 2020. Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. Neurocomputing 398 (2020), 11–19.

[20]

Hao Liu, Jiashi Feng, Meibin Qi, Jianguo Jiang, and Shuicheng Yan. 2017. End-to-end comparative attention networks for person re-identification. IEEE Trans. Image Process. 26, 99 (2017), 3492–3506.

Digital Library

[21]

Yan Lu, Yue Wu, Bin Liu, Tianzhu Zhang, Baopu Li, Qi Chu, and Nenghai Yu. 2020. Cross-modality person re-identification with shared-specific feature transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE.

[22]

Ye Mang, Shen Jianbing, Lin Gaojie, Xiang Tao, Shao Ling, and Steven C. H. Hoi. 2021. Deep learning for person re-identification: a survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence PP, 99 (2021), 1–1.

[23]

Ye Mang, Lan Xiangyuan, and Leng Qingming. 2019. Modality-aware Collaborative Learning for Visible Thermal Person Re-Identification. In 27th ACM International Conference. ACM.

Digital Library

[24]

Xuelin Qian, Yanwei Fu, Wenxuan Wang, Tao Xiang, Yang Wu, Yu-Gang Jiang, and Xiangyang Xue. 2018. Pose-normalized image generation for person re-identification. In Proceedings of the 15th European Conference, Munich, Germany. Springer, Cham.

[25]

Ergys Ristani and Carlo Tomasi. 2018. Features for multi-target multi-camera tracking and re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6036–6046.

[26]

Hailin Shi, Yang Yang, Xiangyu Zhu, Shengcai Liao, Zhen Lei, and Stan Z. Li. 2016. Embedding deep metric for person re-identification: A study against large variations. In European Conference on Computer Vision. Springer, Cham, 732–748.

[27]

Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc., 1857–1865.

Digital Library

[28]

Rahul Rama Varior, Mrinal Haloi, and Gang Wang. 2016. Gated siamese convolutional neural network architecture for human re-identification. In Proceedings of the European Conference on Computer Vision.

[29]

Rahul Rama Varior, Bing Shuai, Jiwen Lu, Dong Xu, and Gang Wang. 2016. A siamese long short-term memory architecture for human re-identification. In Proceedings of the European Conference on Computer Vision.

[30]

Wang Guan’An, Zhang Tianzhu, Yang Yang, Cheng Jian, Chang Jianlong, Liang Xu, and Hou Zengguang. 2020. Cross-modality paired-images generation for RGB-infrared person re-identification. Proceedings of the AAAI Conference on Artificial Intelligence 34, 7 (2020), 12144–12151.

[31]

Yicheng Wang, Zhenzhong Chen, Feng Wu, and Gang Wang. 2018. Person re-identification with cascaded pairwise convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Reconigtion.

[32]

Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. 2018. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 79–88.

[33]

Ancong Wu, Wei Shi Zheng, Hong Xing Yu, Shaogang Gong, and Jianhuang Lai. 2017. RGB-infrared cross-modality person re-identification. In Proceedings of the International Conference on Computer Vision (ICCV’17). IEEE, Los Alamitos, CA.

[34]

Mang Ye, Xiangyuan Lan, Jiawei Li, and Pong Yuen. 2018. Hierarchical discriminative learning for visible thermal person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI.

[35]

Mang Ye, Zheng Wang, Xiangyuan Lan, and Pong C. Yuen. 2018. Visible thermal person re-identification via dual-constrained top-ranking. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI’18).

Digital Library

[36]

Shizhou Zhang, Yifei Yang, Peng Wang, Xiuwei Zhang, and Yanning Zhang. 2021. Attend to the difference: Cross-modality person re-identification via contrastive correlation. IEEE Transactions on Image Processing 30 (2021), 8861–8872.

[37]

Yun Bo Zhao, Jian Wu Lin, Qi Xuan, and Xugang Xi. 2020. HPILN: A feature learning framework for cross-modality person re-identification. IET Image Process. 13, 14 (2020), 2897–2904. DOI:

[38]

Liang Zheng, Yi Yang, and Alexander G. Hauptmann. 2016. Person re-identification: Past, present and future.

[39]

Wang Zhixiang, Wang Zheng, Zheng Yinqiang, Chuang Yung Yu, and Satoh Shin’Ich. 2019. Learning to reduce dual-level discrepancy for infrared-visible person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19).

[40]

Zhun Zhong, Liang Zheng, Zhedong Zheng, Shaozi Li, and Yi Yang. 2018. Camera style adaptation for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Cited By

Zhu YZheng YLiu JLi YZha Z(2024)Noise-Resistance Learning via Multi-Granularity Consistency for Unsupervised Domain Adaptive Person Re-IdentificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/370232821:1(1-23)Online publication date: 2-Nov-2024
https://dl.acm.org/doi/10.1145/3702328
Wu TZhang SChen DHu H(2024)Text-and-Image Learning Transformer for Cross-Modal Person Re-IdentificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368616021:1(1-18)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3686160
Yuan BLu JYou SBao B(2024)Unbiased Feature Learning with Causal Intervention for Visible-Infrared Person Re-IdentificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367473720:10(1-20)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3674737
Show More Cited By

Index Terms

Hybrid Modality Metric Learning for Visible-Infrared Person Re-Identification
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Biometrics

Recommendations

Class-Aware Modality Mix and Center-Guided Metric Learning for Visible-Thermal Person Re-Identification
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Visible thermal person re-identification (VT-REID) is an important and challenging task in that 1) weak lighting environments are inevitably encountered in real-world settings and 2) the inter-modality discrepancy is serious. Most existing methods ...
Visible–infrared person re-identification via patch-mixed cross-modality learning
Abstract
Visible–infrared person re-identification (VI-ReID) aims to retrieve images of the same pedestrian from different modalities, where the challenges lie in the significant modality discrepancy. To alleviate the modality gap, recent methods generate ...
Highlights
- Our method treats the RGB and IR images in the same way and alleviate the modality imbalance problem in VI-ReID.
- The part-alignment loss constrains the consistency of part and global prediction distributions.
- The patch-mixed ...
Discovering attention-guided cross-modality correlation for visible–infrared person re-identification
Abstract
Visible–infrared person re-identification (VI Re-ID) is an essential and challenging task. Existing studies mainly focus on learning the unified modality-invariant representations directly from visible and infrared images. However, it is hard to ...
Graphical abstract

Display Omitted
Highlights
- A novel attention-guided cross-modality correlation approach for VI Re-ID.
- A modality-aware attention mechanism is utilized to mine modality-shared regions and discriminative patterns.
- An attention-guided channel and spatial ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 18, Issue 1s

February 2022

352 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3505206

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 January 2022

Accepted: 01 June 2021

Revised: 01 May 2021

Received: 01 January 2020

Published in TOMM Volume 18, Issue 1s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

Key-Area Research and Development Program of Guangdong Province
National Natural Science Foundation of China
Open Project of Key Laboratory of Ministry of Public Security for Road Traffic Safety

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
674
Total Downloads

Downloads (Last 12 months)101
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhu YZheng YLiu JLi YZha Z(2024)Noise-Resistance Learning via Multi-Granularity Consistency for Unsupervised Domain Adaptive Person Re-IdentificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/370232821:1(1-23)Online publication date: 2-Nov-2024
https://dl.acm.org/doi/10.1145/3702328
Wu TZhang SChen DHu H(2024)Text-and-Image Learning Transformer for Cross-Modal Person Re-IdentificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368616021:1(1-18)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3686160
Yuan BLu JYou SBao B(2024)Unbiased Feature Learning with Causal Intervention for Visible-Infrared Person Re-IdentificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367473720:10(1-20)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3674737
Liu ZYang YWu KLiu QXu XMa XTang J(2024)ASIFusion: An Adaptive Saliency Injection-Based Infrared and Visible Image Fusion NetworkACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366589320:9(1-23)Online publication date: 23-May-2024
https://dl.acm.org/doi/10.1145/3665893
Zeng XWang XXie Y(2024)Multiple Pseudo-Siamese Network with Supervised Contrast Learning for Medical Multi-modal RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363744120:5(1-23)Online publication date: 11-Jan-2024
https://dl.acm.org/doi/10.1145/3637441
Liu HWu JLi FJiang JHong R(2024)SYRER: Synergistic Relational Reasoning for RGB-D Cross-Modal Re-IdentificationIEEE Transactions on Multimedia10.1109/TMM.2023.333805826(5600-5614)Online publication date: 2024
https://doi.org/10.1109/TMM.2023.3338058
Li ZLiu HPeng XJiang W(2024)Inter-Intra Modality Knowledge Learning and Clustering Noise Alleviation for Unsupervised Visible-Infrared Person Re-IdentificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.336730436:8(3934-3947)Online publication date: Aug-2024
https://doi.org/10.1109/TKDE.2024.3367304
Zhang GZhang YZhang HChen YZheng Y(2024)Learning dual attention enhancement feature for visible–infrared person re-identificationJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.10407699:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jvcir.2024.104076
Ning EWang CZhang HNing XTiwari P(2024)Occluded person re-identification with deep learningExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122419239:COnline publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122419
Zhang LZhao XDu HSun JWang J(2024)Learning enhancing modality-invariant features for visible-infrared person re-identificationInternational Journal of Machine Learning and Cybernetics10.1007/s13042-024-02168-6Online publication date: 22-Apr-2024
https://doi.org/10.1007/s13042-024-02168-6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents