Skip to main content
SpringerLink
Log in

Dual-Level Information Transfer for Visible-Thermal Person Re-identification

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Visible-thermal person re-identification (VT-ReID) is a challenging pedestrian retrieval problem in the field of security. Due to the intra-modality variations and cross-modality discrepancy caused by different spectrums, it is difficult to extract discriminative features. Existing works are devoted to projecting different-modality features into a shared space, which has weak discriminability and ignores the contextual relationship. In this paper, a novel dual-level information transfer framework is proposed to reduce the modality discrepancy in image level and feature level for VT-ReID. An auxiliary mix-modality is proposed and a mix-visible-thermal (MVT) learning strategy is built to reduce the discrepancy in image level. Firstly, the mix-modality is generated by a mixup scheme which alleviates the direct transfer. Secondly, under the MVT framework, we use ID loss and hetero center triplet loss to guide feature extraction for visible, thermal, and mixed modalities on a one-stream Network. To enhance the robustness of feature extraction, we introduce a graph information transfer module to transfer information across intra-modality and inter-modality in feature level. We build the agent node for modality by using the modality center, where the agent node aggregates the information of all samples in one modality, and then the information from one modality is transmitted to other modalities through the agent nodes. Extensive experimental results on SYSU-MM01 and RegDB datasets show that our method achieves excellent performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Zheng L, Yang Y, Hauptmann AG (2016) Person Re-identification: Past. Present Future 14(8):1–20

    Google Scholar 

  2. Dong H, Lu P, Zhong S, Liu C, Ji Y, Gong S (2018) Person re-identification by enhanced local maximal occurrence representation and generalized similarity metric learning. Neurocomputing 307:25–37. https://doi.org/10.1016/j.neucom.2018.04.013

    Article  Google Scholar 

  3. Chen D, Wu P, Jia T, Xu F (2022) HOB-net: high-order block network via deep metric learning for person re-identification. Appl Intell 52:4844–4857. https://doi.org/10.1007/S10489-021-02450-Y/TABLES/4

    Article  Google Scholar 

  4. Yao H, Zhang S, Hong R, Zhang Y, Xu C, Tian Q (2019) Deep Representation Learning With Part Loss for Person Re-Identification. IEEE Trans Image Process 28(6):2860–2871. https://doi.org/10.1109/TIP.2019.2891888

    Article  MathSciNet  MATH  Google Scholar 

  5. Zhao Y-B, Lin J-W, Xuan Q, Xi X (2019) HPILN: a feature learning framework for cross-modality person re-identification. IET Image Proc 13(14):2897–2904. https://doi.org/10.1049/iet-ipr.2019.0699

    Article  Google Scholar 

  6. Gao G, Shao H, Wu F, Yang M, Yu Y (2021) Leaning compact and representative features for cross-modality person re-identification. World Wide Web. https://doi.org/10.1007/s11280-022-01014-5

    Article  Google Scholar 

  7. Xiang X, Lv N, Zhai M, Abdeen R, El Saddik A (2020) Dual-path part-level method for visible-infrared person re-identification. Neural Process Lett 52(1):313–328. https://doi.org/10.1007/s11063-020-10239-2

    Article  Google Scholar 

  8. Wu A, Zheng W-S, Yu H-X, Gong S, Lai J (2017) RGB-infrared cross-modality person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 5390–5399. https://doi.org/10.1109/ICCV.2017.575

  9. Ye M, Wang Z, Lan X, Yuen PC (2018) Visible thermal person re-identification via dual-constrained top-ranking. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, California, pp 1092–1099. https://doi.org/10.24963/ijcai.2018/152

  10. Liu H, Tan X, Zhou X (2021) Parameter sharing exploration and hetero-center triplet loss for visible-thermal person re-identification. IEEE Trans Multimed 23:4414–4425. https://doi.org/10.1109/TMM.2020.3042080

    Article  Google Scholar 

  11. Fan X, Luo H, Zhang C, Jiang W (2020) Cross-Spectrum Dual-Subspace Pairing for RGB-infrared Cross-Modality Person Re-Identification 14(8):1–13. arXiv preprint arXiv:2003.00213

  12. Zhu Y, Yang Z, Wang L, Zhao S, Hu X, Tao D (2020) Hetero-Center loss for cross-modality person Re-identification. Neurocomputing 386:97–109. https://doi.org/10.1016/j.neucom.2019.12.100

    Article  Google Scholar 

  13. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Proceedings of the advances in neural information processing systems(NeurIPS), vol 27, pp 2672–2680. https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

  14. Wang Z, Wang Z, Zheng Y, Chuang YY, Satoh S (2019) Learning to reduce dual-level discrepancy for infrared-visible person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 2019-June, pp 618–626. https://doi.org/10.1109/CVPR.2019.00071

  15. Zheng Z, Yang X, Yu Z, Zheng L, Yang Y, Kautz J (2019) Joint Discriminative and Generative Learning for Person Re-Identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 2133–2142. https://doi.org/10.1109/CVPR.2019.00224

  16. Zhang Z, Jiang S, Huang C, Li Y, Xu RYD (2021) RGB-IR cross-modality person ReID based on teacher-student GAN model. Pattern Recogn Lett 150:155–161. https://doi.org/10.1016/j.patrec.2021.07.006

    Article  Google Scholar 

  17. Zhang H, Cisse M, Dauphin YN, Lopez-Paz D (2017) mixup: Beyond empirical risk minimization. In: Proceedings of the 6th international conference on learning representations, ICLR 2018—conference track proceedings. arXiv:1710.09412

  18. Cheng D, Gong Y, Zhou S, Wang J, Zheng N (2016) Person re-identification by multi-channel parts-based CNN with improved triplet loss function. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 1335–1344. https://doi.org/10.1109/CVPR.2016.149

  19. Liu H, Cheng J, Wang W, Su Y, Bai H (2020) Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. Neurocomputing 398:11–19. https://doi.org/10.1016/j.neucom.2020.01.089

    Article  Google Scholar 

  20. Ye M, Shen J, Lin G, Xiang T, Shao L, Hoi SCH (2021) Deep learning for person re-identification: a survey and outlook. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3054775

    Article  Google Scholar 

  21. Yang X, Liu W, Liu W, Tao D (2021) A survey on canonical correlation analysis. IEEE Trans Knowl Data Eng 33(6):2349–2368. https://doi.org/10.1109/TKDE.2019.2958342

    Article  Google Scholar 

  22. Pedagadi S, Orwell J, Velastin S, Boghossian B (2013) Local fisher discriminant analysis for pedestrian re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 3318–3325. https://doi.org/10.1109/CVPR.2013.426

  23. Yang Y, Yang J, Yan J, Liao S, Yi D, Li SZ (2014) Salient color names for person re-identification. Proc Lect Notes Comput Sci 8689:536–551. https://doi.org/10.1007/978-3-319-10590-1_35

    Article  Google Scholar 

  24. Liu C, Gong S, Loy CC, Lin X (2012) Person re-identification: What features are important? Proc Lect Notes Comput Sci 7583:391–401. https://doi.org/10.1007/978-3-642-33863-2_39

    Article  Google Scholar 

  25. Matsukawa T, Okabe T, Suzuki E, Sato Y (2016) Hierarchical Gaussian descriptor for person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 1363–1372. https://doi.org/10.1109/CVPR.2016.152

  26. An L, Chen X, Yang S, Li X (2017) Person re-identification by multi-hypergraph fusion. IEEE Trans Neural Netw Learn Syst 28(11):2763–2774. https://doi.org/10.1109/TNNLS.2016.2602082

    Article  MathSciNet  Google Scholar 

  27. Cheng DS, Cristani M, Stoppa M, Bazzani L, Murino V (2011) Custom pictorial structures for re-identification. In: Procedings of the British machine vision conference 2011, pp 68–16811. https://doi.org/10.5244/C.25.68

  28. Luo H, Jiang W, Zhang X, Fan X, Qian J, Zhang C (2019) AlignedReID++: dynamically matching local information for person re-identification. Pattern Recogn 94:53–61. https://doi.org/10.1016/j.patcog.2019.05.028

    Article  Google Scholar 

  29. Li D, Hu R, Huang W, Li D, Wang X, Hu C (2021) Trajectory association for person re-identification. Neural Process Lett 53(5):3267–3285. https://doi.org/10.1007/s11063-021-10540-8

    Article  Google Scholar 

  30. Chen Y, Zhu X, Gong S (2017) Person re-identification by deep learning multi-scale representations. In: Proceedings of the IEEE international conference on computer vision workshops (ICCVW), pp 2590–2600. https://doi.org/10.1109/ICCVW.2017.304

  31. Luo C, Song C, Zhang Z (2020) Generalizing person re-identification by camera-aware invariance learning and cross-domain mixup. Lect Notes Comput Sci 12360:224–241. https://doi.org/10.1007/978-3-030-58555-6_14

    Article  Google Scholar 

  32. Sun Y, Zheng L, Yang Y, Tian Q, Wang S (2018) Beyond part models: person retrieval with refined part pooling (and a strong convolutional baseline). Lect Notes Comput Sci 11208:501–518. https://doi.org/10.1007/978-3-030-01225-0_30

    Article  Google Scholar 

  33. Wang H, Bi X (2021) Person re-identification based on graph relation learning. Neural Process Lett 53:1401–1415. https://doi.org/10.1007/s11063-021-10446-5

    Article  Google Scholar 

  34. Su C, Li J, Zhang S, Xing J, Gao W, Tian Q (2017) Pose-driven deep convolutional model for person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 3980–3989. https://doi.org/10.1109/ICCV.2017.427

  35. Zhang Z, Lan C, Zeng W, Chen Z (2019) Densely semantically aligned person re-identification. In: Proceedings of the IEEE Computer society conference on computer vision and pattern recognition (CVPR), pp 667–676. https://doi.org/10.1109/CVPR.2019.00076

  36. Liu X, Zhao H, Tian M, Sheng L, Shao J, Yi S, Yan J, Wang X (2017) HydraPlus-Net: attentive deep features for pedestrian analysis. In: Proceedings of the international conference on computer vision (ICCV), pp 350–359. https://doi.org/10.1109/ICCV.2017.46

  37. Ning X, Gong K, Li W, Zhang L, Bai X, Tian S (2021) Feature refinement and filter network for person re-identification. IEEE Trans Circuits Syst Video Technol 31(9):3391–3402. https://doi.org/10.1109/TCSVT.2020.3043026

    Article  Google Scholar 

  38. Yang F, Yan K, Lu S, Jia H, Xie X, Gao W (2019) Attention driven person re-identification. Pattern Recogn 86:143–155. https://doi.org/10.1016/j.patcog.2018.08.015

    Article  Google Scholar 

  39. Zhang M, Xin M, Gao C, Wang X, Zhang S (2020) Attention-aware scoring learning for person re-identification. Knowl-Based Syst 203:106154. https://doi.org/10.1016/j.knosys.2020.106154

    Article  Google Scholar 

  40. Wang X, Gao C, Xin M, Zhang S, Zhang M (2021) Topology and channel affinity reinforced global attention for person re-identification. Int J Intell Syst 36(9):5136–5160. https://doi.org/10.1002/int.22506

    Article  Google Scholar 

  41. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th international conference on learning representations, ICLR 2017—conference track proceedings, pp 1–14. arXiv:1609.02907

  42. Zhang S, Yang Y, Wang P, Liang G, Zhang X, Zhang Y (2021) Attend to the difference: cross-modality person re-identification via contrastive correlation. IEEE Trans Image Process 30(8):8861–8872. https://doi.org/10.1109/TIP.2021.3120881

    Article  Google Scholar 

  43. Ye M, Lan X, Li J, Yuen PC (2018) Hierarchical discriminative learning for visible thermal person re-identification. In: Proceedings of the 32nd AAAI conference on artificial intelligence, AAAI 2018, pp 7501–7508. https://ojs.aaai.org/index.php/AAAI/article/view/12293

  44. Liu Q, Teng Q, Chen H, Li B, Qing L (2022) Dual adaptive alignment and partitioning network for visible and infrared cross-modality person re-identification. Appl Intell 52(1):547–563. https://doi.org/10.1007/S10489-021-02390-7/FIGURES/11

    Article  Google Scholar 

  45. Bai Z, Wang Z, Wang J, Hu D, Ding E (2021) Unsupervised multi-source domain adaptation for person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 12909–12918. https://doi.org/10.1109/CVPR46437.2021.01272

  46. Liu W, Li J, Liu B, Guan W, Zhou Y, Xu C (2021) Unified cross-domain classification via geometric and statistical adaptations. Pattern Recogn 110:107658. https://doi.org/10.1016/j.patcog.2020.107658

    Article  Google Scholar 

  47. Zhang B, Xiao J, Jiao J, Wei Y, Zhao Y (2021) Affinity attention graph neural network for weakly supervised semantic segmentation. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3083269

    Article  Google Scholar 

  48. Li W, Qi K, Chen W, Zhou Y (2021) Unified batch all triplet loss for visible-infrared person re-identification. In: Proceedings of the 2021 international joint conference on neural networks (IJCNN), pp 1–8. https://doi.org/10.1109/IJCNN52387.2021.9533325

  49. Hao Y, Wang N, Li J, Gao X (2019) HSME: Hypersphere manifold embedding for visible thermal person re-identification. Proc AAAI Conf Artif Intell 33:8385–8392. https://doi.org/10.1609/aaai.v33i01.33018385

    Article  Google Scholar 

  50. Dai P, Ji R, Wang H, Wu Q, Huang Y (2018) Cross-modality person re-identification with generative adversarial training. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, California, pp 677–683. https://doi.org/10.24963/ijcai.2018/94

  51. Wang G, Zhang T, Cheng J, Liu S, Yang Y, Hou Z (2019) RGB-infrared cross-modality person re-identification via joint pixel and feature alignment. In: Proceedings of the international conference on computer vision (ICCV), pp 3622–3631. https://doi.org/10.1109/ICCV.2019.00372

  52. Choi S, Lee S, Kim Y, Kim T, Kim C (2020) Hi-CMD: hierarchical cross-modality disentanglement for visible-infrared person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 10254–10263. https://doi.org/10.1109/CVPR42600.2020.01027

  53. Kumar K, Shrimankar DD (2018) F-DES: fast and deep event summarization. IEEE Trans Multimed 20(2):323–334. https://doi.org/10.1109/TMM.2017.2741423

    Article  Google Scholar 

  54. Kumar K, Shrimankar DD, Singh N (2018) Eratosthenes sieve based key-frame extraction technique for event summarization in videos. Multimed Tools Appl 77(6):7383–7404

    Article  Google Scholar 

  55. Kumar K, Shrimankar DD (2018) Deep event learning boost-up approach: DELTA. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-5882-z

    Article  Google Scholar 

  56. Kumar A, Singh N, Kumar P, Vijayvergia A, Kumar K (2017) A novel superpixel based color spatial feature for salient object detection. In: 2017 Conference on information and communication technology (CICT), pp 1–5. https://doi.org/10.1109/INFOCOMTECH.2017.8340630

  57. Sharma S, Shivhare SN, Singh N, Kumar K (2019) Computationally efficient ANN model for small-scale problems. Adv Intell Syst Comput. https://doi.org/10.1007/978-981-13-0923-6_37

    Article  Google Scholar 

  58. Kumar K, Mishra A, Dahiya S, Kumar A (2022) A technique for human upper body parts movement tracking. IETE J Res. https://doi.org/10.1080/03772063.2022.2048708

    Article  Google Scholar 

  59. He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  60. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  61. Szegedy C, Wei Liu, Yangqing Jia, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

  62. Deng Jia, Dong Wei, Socher R, Li Li-Jia, Li Kai, Fei-Fei Li (2009) ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 248–255. https://doi.org/10.1109/CVPRW.2009.5206848

  63. Wu Z, Wu Z (2010) Exploration, visualization, and preprocessing of high-dimensional data. Methods Mole Biol 620:267–284. https://doi.org/10.1007/978-1-60761-580-4_8

    Article  Google Scholar 

  64. Hermans A, Beyer L, Leibe B (2017) In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737

  65. Nguyen D, Hong H, Kim K, Park K (2017) Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17(3):605. https://doi.org/10.3390/s17030605

    Article  Google Scholar 

  66. Lu Y, Wu Y, Liu B, Zhang T, Li B, Chu Q, Yu N (2020) Cross-modality person re-identification with shared-specific feature transfer. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR42600.2020.01339

  67. Tian X, Zhang Z, Lin S, Qu Y, Xie Y, Ma L (2021) Farewell to mutual information: variational distillation for cross-modal person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 1522–1531. https://doi.org/10.1109/CVPR46437.2021.00157

  68. Wu Q, Dai P, Chen J, Lin C-W, Wu Y, Huang F, Zhong B, Ji R (2021) Discover cross-modality nuances for visible-infrared person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 4330–4339. https://openaccess.thecvf.com/content/CVPR2021/papers/Wu_Discover_Cross-Modality_Nuances_for_Visible-Infrared_Person_Re-Identification_CVPR_2021_paper.pdf

  69. Ye M, Ruan W, Du B, Shou MZ (2021) Channel augmented joint learning for visible-infrared recognition. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 13547–13556. https://doi.org/10.1109/ICCV48922.2021.01331

  70. Yang M, Huang Z, Hu P, Li T, Lv J, Peng X (2022) Learning with twin noisy labels for visible-infrared person re-identification. Cvpr 2022(1):14308–14317

    Google Scholar 

  71. Zhang Q, Lai C, Liu J, Huang N, Han J (2022) FMCNet : Feature-level modality compensation for visible-infrared person. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 7349–7358. https://openaccess.thecvf.com/content/CVPR2022/html/Zhang_FMCNet_Feature-Level_Modality_Compensation_for_Visible-Infrared_Person_Re-Identification_CVPR_2022_paper.html

  72. Wei Z, Yang X, Wang N, Gao X (2021) Syncretic modality collaborative learning for visible infrared person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 225–234. https://doi.org/10.1109/ICCV48922.2021.00029

  73. Ling Y, Luo Z, Lin Y, Li S (2021) A multi-constraint similarity learning with adaptive weighting for visible-thermal person re-identification. In: Proceedings of the thirtieth international joint conference on artificial intelligence, pp 845–851. https://doi.org/10.24963/ijcai.2021/117

Download references

Acknowledgements

This research was funded by the National Natural Science Foundation of China (62002100), and the Key R &D and Promotion Projects in Henan Province (212102210411).

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Henan University, Zhengzhou, 450046, China

    Junyu Song, Xile Wang, Kaifang Li & Miaohui Zhang

Authors
  1. Junyu Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xile Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kaifang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Miaohui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miaohui Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, J., Wang, X., Li, K. et al. Dual-Level Information Transfer for Visible-Thermal Person Re-identification. Neural Process Lett 55, 7999–8021 (2023). https://doi.org/10.1007/s11063-023-11294-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11294-1

Keywords

Search

Navigation