Skip to main content

Advertisement

SpringerLink
Log in

Cross-modality collaborative learning identified pedestrian

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Cross-modal pedestrian re-identification is a key technology to realize all-weather intelligent video surveillance system. This technology is designed to match the visible light image and infrared image of a pedestrian with a specific identity in a non-overlapping camera scene, so it faces huge intra-class changes and modal differences. Existing methods are difficult to solve these two difficulties, which is largely due to the lack of effective mining of feature discrimination and the full use of multi-source heterogeneous information. In view of the above shortcomings, a refined multi-source feature collaborative network is designed by the collaborative learning method, and multiple complementary features are extracted for information fusion, the learning ability of the network is improved. Multi-scale and multi-level features are extracted from the backbone convolutional network, and the refined feature collaborative learning is realized; the discriminative ability of features is enhanced to deal with intra-class changes. A modal sharing and unique feature collaboration module and a cross-modal human semantic self-supervision module are designed to achieve the purpose of multi-source feature collaborative learning, so as to improve the utilization of multi-source heterogeneous image information, and then modal differences are resolved. The validity and advancement of this method are verified on the SYSU-MM01 and RegDB data sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Song, L.L., Li, B., Zhao, J.Y., et al.: Normality resampling of improved metric learning method for person re-identification. Comput. Eng. Appl. 56(8), 158–165 (2020). https://doi.org/10.3778/j.issn.1002-8331.1904-0235

    Article  Google Scholar 

  2. Fan, C.X., Chen, Y.J., Cao, L., et al.: Person re-identification based on visual perceptual model. Comput. Eng. Appl. 52(6), 156–161 (2016). https://doi.org/10.3778/j.issn.1002-8331.1504-0307

    Article  Google Scholar 

  3. Sun, Y., Zheng, L., Yang, Y., et al.: Beyond part models: person retrieval with refined part pooling (and a strong convolutional baseline). In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 480–496 (2018)

  4. Wang, G., Yuan, Y., Chen, X, et al.: Learning discriminative features with multiple granularities for person re-identification. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 274–282 (2018)

  5. He, L., Liao, X., Liu, W., et al.: Fastreid: a pytorch toolbox for general instance re-identification. arXiv:2006.02631arXiv:2006.02631 (2020).

  6. Zheng, L., Yang, Y., Hauptmann, A. G.: Person re-identification: past, present and future. arXiv:1610.02984 (2016).

  7. Zhu, X., Wu, B., Huang, D., et al.: Fast open-world person re-identification. IEEE Trans. Image Process. 27(5), 2286–2300 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  8. Bai, S., Tang, P., Torr, P. H. S., et al.: Re-ranking via metric fusion for object retrieval and person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 740–749 (2019)

  9. Dai, J., Zhang, P., Wang, D., et al.: Video person re-identification by temporal residual learning. IEEE Trans. Image Process. 28(3), 1366–1377 (2018)

    Article  MathSciNet  Google Scholar 

  10. Hou, R., Ma, B., Chang, H. et al. Interaction-and-aggregation network for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9317–9326 (2019)

  11. Song, C., Huang, Y., Ouyang, W., et al.: Mask- guided contrastive attention model for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1179–1188 (2018)

  12. Sun, Y., Xu, Q., Li, Y., et al.: Perceive where to focus: learning visibility- aware part-level features for partial person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 393–402 (2019)

  13. Yang, W., Huang, H., Zhang, Z., et al.: Towards rich feature discovery with class activation maps augmentation for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1389–1398 (2019)

  14. Liao, S., Hu, Y., Zhu, X., et al.: Person re-identification by local maximal occurrence representation and metric learning. In: Proceedings of the IEEE Conference on Com puter Vision and Pattern Recognition, pp. 2197–2206 (2015)

  15. Liao, S., Li, S. Z.: Efficient psd constrained asymmetric metric learning for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3685–3693 (2015)

  16. Ben, X.Y., Xu, S., Wang, K.J.: Review on pedestrian gait feature expression and recognition. Pattern Recognit. Artif. Intell. 25(1), 71–81 (2012). https://doi.org/10.3969/j.issn.1003-6059.2012.01.010

    Article  Google Scholar 

  17. Zhang, X., Luo, H., Fan, X., et al.: AlignedReID: surpassing human-level performance in person re-identification. arXiv:1711.08184, (2017)

  18. Wu, A., Zheng, W. S., Yu, H. X., et al.: Rgb-infrared cross modality person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5380–5389 (2017)

  19. Zheng, L., Bie, Z., Sun, Y., et al.: Mars: a video benchmark for large-scale person re-identification. In: European Conference on Computer Vision, pp. 868–884. Springer, Cham (2016).

  20. Wu, Y., Lin, Y., Dong, X., et al.: Progressive learning for person re-identification with one example. IEEE Trans. Image Process. 28(6), 2872–2881 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  21. Chen, D., Li, H., Liu, X., et al.: Improving deep visual rep resentation for person re-identification by global and local image- language association. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 54–70 (2018)

  22. Nguyen, D.T., Hong, H.G., Kim, K.W., et al.: Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17(3), 605 (2017)

    Article  Google Scholar 

  23. Ye, M., Lan, X., Wang, Z., et al.: Bi-directional center-constrained top-ranking for visible thermal person re-identification. IEEE Trans. Inf. Forensics Secur. 15, 407–419 (2019)

    Article  Google Scholar 

  24. Dai, P., Ji, R., Wang, H., et al.: Cross-modality person re-identification with generative adversarial training. In: 27th International Joint Conference on Artificial Intelligence, (2018)

  25. Wang, Z. X., Wang, Z., Zheng, Y., et al.: Learning to reduce dual- level discrepancy for infrared- visible person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 618–626

  26. Wang, G., Zhang, T., Cheng, J., et al.: Rgb- infrared cross- modality person re- identification via joint pixel and feature alignment. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3623–3632 (2019)

  27. Zhu, Y., Yang, Z., Wang, L., et al.: Hetero- center loss for cross-modality person re-identification. Neurocomputing 386, 97–109 (2020)

    Article  Google Scholar 

  28. Song, G., Chai, W.: Collaborative learning for deep neural networks. Adv. Neural Inf. Process. Syst. 31, 1832–1841 (2018)

    Google Scholar 

  29. Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 7, pp. 1–9 (2015)

  30. Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997)

    Article  MathSciNet  Google Scholar 

  31. Baxter, J.: Learning internal representations. In: Proceedings of the Eighth Annual Conference on Computational Learning Theory, pp. 311–320 (1995)

  32. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowl edge in a neural network. arXiv:1503.02531, (2015)

  33. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  34. Ye, M., Lan, X., Li, J., et al.: Hierarchical discriminative learning for visible thermal person re-identification. In: Thirty- Second AAAI Conference on Artificial Intelligence, (2018)

  35. Choi, S., Lee, S., Kim, Y., et al.: Hi- CMD: hierarchical cross-modality disentanglement for visible-infrared person re- identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10257–10266 (2020)

  36. Lu, Y., Wu, Y., Liu, B., et al.: Cross-modality person re identification with shared-specific feature transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13379–13389 (2020)

  37. Wang, J., Jiao, S., Li, Y., et al.: Two-stage metric learn ing for cross- modality person re-identification. In: Proceedings of the 5th International Conference on Multi media and Image Processing, pp. 28–32 (2020)

  38. Luo, H., Jiang, W., Gu, Y., et al.: A strong baseline and batch normalization neck for deep person re-identification. IEEE Trans. Multimed. 22(10), 2597–2609 (2019)

    Article  Google Scholar 

  39. Ye, M., Lan, X., Li, J., et al.: Hierarchical discriminative learning for visible thermal person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence, (2018)

  40. Hao, Y., Wang, N., Li, J., et al.: HSME: hypersphere manifold embedding for visible thermal person re-identification. Proc. AAAI Conf. Artif. Intell. 33, 8385–8392 (2019)

    Google Scholar 

  41. Zhao, Y.B., Lin, J.W., Xuan, Q., et al.: HPILN: a feature learning framework for cross-modality person re-identification. IET Image Proc. 13(14), 2897–2904 (2019)

    Article  Google Scholar 

  42. Feng, Z., Lai, J., Xie, X.: Learning modality-specific rep resentations for visible-infrared person re-identification. IEEE Trans. Image Process. 29, 579–590 (2019)

    Article  MATH  Google Scholar 

  43. Ye, M., Shen, J., Lin, G., et al. Deep learning for person re-identification: a survey and outlook. arXiv: 2001.04193, (2020)

  44. Li, D., Wei, X., Hong, X., et al.: Infrared- visible cross modal person re-identification with an X modality. In: 34th AAAI Conference on Artificial Intelligence, pp. 4610–4617 (2020)

  45. Ye, M., Shen, J., Shao, L.: Visible- infrared person re identification via homogeneous augmented tri-modal learning. IEEE Trans. Inf. Forensics Secur. 16, 728–739 (2020)

    Article  Google Scholar 

  46. Jia, M., Zhai, Y., Lu, S., et al.: A similarity inference metric for RGB-infrared cross-modality person re-identification. arXiv: 2007.01504, (2020)

  47. Liu, H., Cheng, J., Wang, W., et al.: Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. Neurocomputing 398, 11–19 (2020)

    Article  Google Scholar 

  48. Ye, M., Lan, X.Y., Leng, Q.M., Shen, J.B.: Cross-modality person re-identification via modality-aware collaborative ensemble learning. IEEE Trans. Image Process. 29, 9387–9399 (2020). https://doi.org/10.1109/TIP.2020.2998275

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by First-class course in Hunan Province project ([2021] 322, No.167); Hunan University Student Innovation and Entrepreneurship Training Program: ([2022] 174, No. 4531 [2021] 197. No. 3281): Teaching Reform Research Proiec: Xiangwalingvuan [2022] No. 64.

Author information

Authors and Affiliations

  1. School of Information and Mechanical Engineering, Hunan International Economics University, Changsha, 410205, Hunan, China

    Xiongjun Wen, Xin Feng, Ping Li & Wenfang Chen

Authors
  1. Xiongjun Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenfang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Feng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, X., Feng, X., Li, P. et al. Cross-modality collaborative learning identified pedestrian. Vis Comput 39, 4117–4132 (2023). https://doi.org/10.1007/s00371-022-02579-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02579-y

Keywords

Search

Navigation