Skip to main content
SpringerLink
Log in

Dual Attention Feature Fusion for Visible-Infrared Object Detection

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2023 (ICANN 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14260))

Included in the following conference series:

  • 1119 Accesses

Abstract

Feature fusion is an essential component of multimodal object detection to exploit the complementary information and common information between multi-source images. When it comes to visible-infrared image pairs, however, the visible images are prone to illumination and visibility and there may be a lot of interference information and little useful information. We suggest performing common feature enhancement and spatial cross attention sequentially to solve this problem. For this purpose, a novel Dual Attention Transformer Feature Fusion (DATFF) module which is designed for feature fusion of intermediate feature maps is proposed. We integrate it into two-stream object detectors and achieve state-of-the-art performance on DroneVehicle and FLIR visible-infrared object detection datasets. Our code is available at https://github.com/a21401624/DATFF.

This research was supported by the National Key Research and Development Program of China under Grant No. 2018AAA0100400, and the National Natural Science Foundation of China under Grants 91538201, and 62076242.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/SunYM2020/UA-CMDet.

References

  1. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  2. Chen, K., Wang, J., Pang, J., et al.: MMDetection: open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)

  3. Chen, Y.-T., Shi, J., Ye, Z., Mertz, C., Ramanan, D., Kong, S.: Multimodal object detection via probabilistic ensembling. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13669, pp. 139–158. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20077-9_9

    Chapter  Google Scholar 

  4. Ding, J., Xue, N., Long, Y., et al.: Learning RoI Transformer for oriented object detection in aerial images. In: CVPR, pp. 2844–2853 (2019)

    Google Scholar 

  5. Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR (2021)

    Google Scholar 

  6. Li, C., et al.: Illumination-aware faster r-cnn for robust multispectral pedestrian detection. Pattern Recogn. 85, 161–171 (2019)

    Article  Google Scholar 

  7. Lin, T.Y., Dollár, P., Girshick, R., et al.: Feature pyramid networks for object detection. In: CVPR, pp. 936–944 (2017)

    Google Scholar 

  8. Liu, J., Zhang, S., Wang, S., et al.: Multispectral deep neural networks for pedestrian detection. In: BMVC, pp. 73.1-73.13 (2016)

    Google Scholar 

  9. Liu, T., Lam, K.M., Zhao, R., Qiu, G.: Deep cross-modal representation learning and distillation for illumination-invariant pedestrian detection. IEEE Trans. Circuits Syst. Video Technol. 32(1), 315–329 (2022)

    Article  Google Scholar 

  10. Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  11. Qingyun, F., Dapeng, H., Zhaokui, W.: Cross-modality fusion transformer for multispectral object detection. arXiv preprint arXiv:2111.00273 (2021)

  12. Qingyun, F., Zhaokui, W.: Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery. Pattern Recogn. 130, 108786 (2022)

    Article  Google Scholar 

  13. Razakarivony, S., Jurie, F.: Vehicle detection in aerial imagery: a small target detection benchmark. J. Vis. Commun. Image Represent. 34, 187–203 (2016)

    Article  Google Scholar 

  14. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS, vol. 28, pp. 91–99 (2015)

    Google Scholar 

  15. Sharma, M., Dhanaraj, M., Karnam, S., et al.: YOLOrs: object detection in multimodal remote sensing imagery. IEEE J. Selected Topics Appli. Earth Observat. Remote Sensing 14, 1497–1508 (2021)

    Article  Google Scholar 

  16. Sun, Y., Cao, B., Zhu, P., et al.: Drone-based RGB-Infrared cross-modality vehicle detection via uncertainty-aware learning. IEEE Trans. Circuits Syst. Video Technol. 32(10), 6700–6713 (2022)

    Article  Google Scholar 

  17. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: NIPS, vol. 30, pp. 5998–6008 (2017)

    Google Scholar 

  18. Xie, X., Cheng, G., Wang, J., Yao, X., Han, J.: Oriented R-CNN for object detection. In: ICCV, pp. 3500–3509 (2021)

    Google Scholar 

  19. Yang, X., Yan, J., Feng, Z., He, T.: R3det: refined single-stage detector with feature refinement for rotating object. In: AAAI, vol. 35(4), pp. 3163–3171 (2021)

    Google Scholar 

  20. Zhang, H., Fromont, E., Lefevre, S., et al.: Guided attentive feature fusion for multispectral pedestrian detection. In: WACV, pp. 72–80 (2021)

    Google Scholar 

  21. Zhang, H., Fromont, E., et al.: Multispectral fusion for object detection with cyclic fuse-and-refine blocks. In: ICIP, pp. 276–280 (2020)

    Google Scholar 

  22. Zhang, J., Lei, J., Xie, W., et al.: SuperYOLO: Super resolution assisted object detection in multimodal remote sensing imagery. IEEE Trans. Geosci. Remote Sens. 61, 1–15 (2023)

    Google Scholar 

  23. Zhang, L., Liu, Z., Zhang, S., Yang, X., et al.: Cross-modality interactive attention network for multispectral pedestrian detection. Inform. Fusion 50, 20–29 (2019)

    Article  Google Scholar 

  24. Zhang, X., Jiang, H., Xu, N., et al.: MsIFT: multi-source image fusion transformer. Remote Sensing 14(16) (2022)

    Google Scholar 

  25. Zhou, K., Chen, L., Cao, X.: Improving multispectral pedestrian detection by addressing modality imbalance problems. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12363, pp. 787–803. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58523-5_46

    Chapter  Google Scholar 

  26. Zhou, Y., Yang, X., Zhang, G., et al.: MMRotate: a rotated object detection benchmark using pytorch. arXiv preprint arXiv:2204.13317 (2022)

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Yuxuan Hu

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

    Yuxuan Hu

  3. Research Center of Aerospace Information, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Limin Shi & Lubin Weng

  4. Institute of Information Fusion, Naval Aviation University, Yantai, China

    Libo Yao

Authors
  1. Yuxuan Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Limin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Libo Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lubin Weng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lubin Weng .

Editor information

Editors and Affiliations

  1. Democritus University of Thrace, Xanthi, Greece

    Lazaros Iliadis

  2. Democritus University of Thrace, Xanthi, Greece

    Antonios Papaleonidas

  3. Lancaster University, Lancaster, UK

    Plamen Angelov

  4. Teesside University, Middlesbrough, UK

    Chrisina Jayne

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hu, Y., Shi, L., Yao, L., Weng, L. (2023). Dual Attention Feature Fusion for Visible-Infrared Object Detection. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial Neural Networks and Machine Learning – ICANN 2023. ICANN 2023. Lecture Notes in Computer Science, vol 14260. Springer, Cham. https://doi.org/10.1007/978-3-031-44195-0_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-44195-0_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-44194-3

  • Online ISBN: 978-3-031-44195-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation