Skip to main content
SpringerLink
Log in

Cross-modal identity correlation mining for visible-thermal person re-identification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Visible-thermal person recognition is a sub problem of image retrieval, which aims to find out the images belonging to the same pedestrian as the current image from the image set of another modality. In this paper, we propose a novel cross-modal identity correlation mining algorithm to mine potential correlation knowledge from the features of visible and thermal modalities. First, aiming at the huge visual differences caused by different imaging mechanisms, we build a correlation-enhanced knowledge transfer module based on cross-modal identity similarity to enhance the feature representation by exchanging identity knowledge between two modalities and then compress it into a shared subspace. Second, in view of different pedestrian posture and camera perspective, we design a symmetric modal-specific feature embedding module to improve the intra-modality feature discrimination, which maps the two modal images to a pair of independent feature subspaces by two fine-grained network branches. The whole algorithm can be trained in an end-to-end manner. Extensive experiments demonstrated that the proposed method outperforms the state-of-the-art methods on SYSU-MM01 and RegDB.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Arora M, Kumar M, Garg NK (2018) Facial Emotion Recognition System Based on PCA and Gradient Features. National Academy Science Letters 41.6: https://doi.org/10.1007/s40009-018-0694-2

  2. Bansal M et al (2020) An efficient technique for object recognition using Shi-Tomasi corner detection algorithm. Soft Computing 3.prepublish: https://doi.org/10.1007/s00500-020-05453-y

  3. Basaran E, Gokmen M, Kamasak ME (2019) An efficient framework for visible-infrared cross modality person re-identification. In: arXiv preprint, arXiv:1907.06498

  4. Chen B, Deng W, Hu J (2019) Mixed high-order attention network for person re-identification. In: International conference on computer vision (ICCV)

  5. Dai P, Ji R, Wang H, Wu Q, Huang Y (2018) Cross-modality person re-identification with generative adversarial training. In: IJCAI, pp 677–683

  6. Dargan S, Kumar M (2019) Writer Identification System for Indic and Non-Indic Scripts: State-of-the-Art Survey. Archives of Computational Methods in Engineering 26.4: https://doi.org/10.1007/s11831-018-9278-z

  7. Feng Z, Lai J, Xie X (2020) Learning modality-specific representations for visible-infrared person re-identification. In: IEEE TIP

  8. Gao S et al (2020) Pose-guided visible part matching for occluded person ReID. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  9. Gupta S et al (2019) Improved object recognition results using SIFT and ORB feature detector. Multimedia Tools and Applications 78.23: https://doi.org/10.1007/s11042-019-08232-6

  10. Hao Y, Wang N, Li J, Gao X (2019) Hsme: Hypersphere manifold embedding for visible thermal person re-identification. In: Proceedings of the AAAI conference on artificial intelligence, pp 8385–8392

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778

  12. Hermans A, Beyer L, Leibe B (2017) In defense of the triplet loss for person re-identification. In: IEEE international conference on computer vision(ICCV)

  13. Hinton GE, Vinyals O, Dean J (2015) Distilling the Knowledge in a Neural Network. In: arXiv preprint, arXiv:1503.02531

  14. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  15. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML, pp 448–456

  16. Kumar M et al (2020) Performance evaluation of classifiers for the recognition of offline handwritten Gurmukhi characters and numerals: a study. Artificial Intelligence Review: An International Science and Engineering Journal 53.1: https://doi.org/10.1007/s10462-019-09727-2

  17. Li D, Wei X, Hong X, Gong Y (2020) Infrared-visible cross-modal person re-identification with an x modality. In: The Association for the advance of artificial intelligence(AAAI)

  18. Li J et al (2019) Global-local temporal representations for video person re-identification. In: IEEE/CVF international conference on computer vision (ICCV)

  19. Liang J et al (2019) Related attention network for person re-identification. In: IEEE Fifth international conference on multimedia big data (BigMM)

  20. Liu X, Zhang S, Yang M (2019) Self-guided hash coding for large-scale person re-identification. In: IEEE conference on multimedia information processing and retrieval (MIPR)

  21. Liu H et al (2020) Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. In: Neurocomputing

  22. Lu Y et al (2020) Cross-Modality Person Re-Identification with Shared-Specific feature transfer. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), seattle, WA, USA, pp 13376–13386

  23. Luo H, Gu Y, Liao X, Lai S, Jiang W (2019) Bag of tricks and a strong baseline for deep person re-identification. In: IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 1487–1495

  24. Monika B, Munish K, Manish K (2020) 2D Object Recognition Techniques: State-of-the-Art Work. Archives of Computational Methods in Engineering 28.3: https://doi.org/10.1007/S11831-020-09409-1

  25. Nguyen DT, Hong HG, Kim KW, Park KR (2017) Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17:605

    Article  Google Scholar 

  26. Peng B, Jin X, Liu J, Li D, Wu Y, Liu Y, Zhang Z (2019) Correlation congruence for knowledge distillatio. In: Proceedings of the IEEE international conference on computer vision(ICCV)

  27. Quan R, Dong X, Wu Y, Zhu L, Yang Y (2019) Auto-reID: Searching for a Part-Aware ConvNet for person re-identification. In: International conference on computer vision (ICCV), pp 3749–3758

  28. Shen Y, Li H, Yi S, Chen D, Wang X (2018) Person re-identification with deep similarity-guided graph neural network. In: Proceedings of the European conference on computer vision (ECCV), pp 486– 504

  29. Sun Y, Zheng L, Yang Y, Tian Q, Wang S (2018) Beyond Part models: Person retrieval with refined part pooling and a strong convolutional baseline. In: European conference on computer vision(ECCV)

  30. Surbhi G, Kutub T, Munish K (2020) 2D-human face recognition using SIFT and SURF descriptors of face’s feature regions. The Visual Computer 37.3: https://doi.org/10.1007/S00371-020-01814-8

  31. Varior RR, Haloi M, Wang G (2016) Gated siamese convolutional neural network architecture for human re-identification. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 791–808

  32. Wang Z, Wang Z, Zheng Y, Chuang Y-Y, Satoh S (2019) Learning to reduce dual-level discrepancy for infrared-visible person reidentification. In: CVPR, pp 618–626

  33. Wang G, Zhang T, Cheng J, Si L, Yang Y, Hou Z (2019) Rgb-infrared cross-modality person re-identification via joint pixel and feature alignment. In: Proceedings of the IEEE international conference on computer vision(ICCV), pp 3623–3632

  34. Wu A, Zheng W-S, Yu H-X, Gong S, Lai J (2017) Rgb-infrared cross-modality person re-identification. In: Proceedings of the IEEE international conference on computer vision(ICCV), pp 5380–5389

  35. Ye M, Lan X, Leng Q, Shen J (2020) Cross-modality person re-identification via modality-aware collaborative ensemble learning. In: IEEE transactions on image processing(TIP)

  36. Ye M, Shen J, jie Lin G, Xiang T, Shao L, Hoi SCH (2020) Deep learning for person re-identification: A survey and outlook. In: arXiv preprint, arXiv:2001.04193

  37. Zhang L, Song J, Gao A, Chen J, Bao C, Ma K (2019) Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In: Proceedings of the IEEE international conference on computer vision(ICCV)

  38. Zhang Y, Xiang T, Hospedales TM, Lu H (2018) Deep mutual learning. In: Conference on computer vision and pattern recognition (CVPR)

  39. Zhao Y-B, Lin J-W, Xuan Q, Xi X (2019) Hpiln: A feature learning framework for cross-modality person re-identification. IET Image Process 13(14):2897–2904

    Article  Google Scholar 

  40. Zhong X et al (2020) Visible-infrared person re-identification via colorization-based siamese generative adversarial network. In: International conference on multimedia retrieval (ICMR)

  41. Zhou Guorui et al (2017) Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net. arXiv:1708.04106

  42. Zhu X, Morerio P, Murino V (2019) Unsupervised domain-adaptive person re-identification based on attributes. In: IEEE international conference on image processing (ICIP), pp 4110–4114

Download references

Acknowledgements

The National Natural Science Foundation of China (Grant No. 62176027 and 62102179); the General Program of National Natural Science Foundation of Chongqing (Grant No. cstc2020jcyj-msxmX0790); Smart Community Project Based on Machine Vision and Internetof Things Platform (Grant No. ZH22017002200003PWC).

Author information

Authors and Affiliations

  1. School of Computer Science, Chongqing University, Chongqing, 400044, China

    Sen Zhang, Zhaowei Shang & Mingliang Zhou

  2. National Engineering Laboratory for Dangerous Articles and Explosives Detection Technologies, Department of Engineering Physics, Tsinghua University, Beijing, 100084, China

    Yingxin Wang

  3. Suzhou Automotive Research Institute of Tsinghua University, Suzhou, 215131, China

    Guoliang Sun

Authors
  1. Sen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhaowei Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingliang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yingxin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guoliang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhaowei Shang or Mingliang Zhou.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Shang, Z., Zhou, M. et al. Cross-modal identity correlation mining for visible-thermal person re-identification. Multimed Tools Appl 81, 39981–39994 (2022). https://doi.org/10.1007/s11042-022-13090-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13090-w

Keywords

Search

Navigation