Skip to main content
SpringerLink
Log in

LGF\(^2\): Local and Global Feature Fusion for Text-Guided Object Detection

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2023 (ICANN 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14260))

Included in the following conference series:

  • 1025 Accesses

Abstract

A baby can successfully learn to identify objects in an image with the corresponding text description provided by their parents. This learning process leverages the multimodal information of both the image and text. However, classical object detection approaches only utilize the image modality to distinguish objects, neglecting the text modality. While many Vision-Language Models have been explored in the object detection task, they often require large amounts of pre-training data and can only specify a particular model structure. In this paper, we propose a lightweight and generic Local and Global Feature Fusion (LGF\(^2\)) framework for text-guided object detection to enhance the performance of image modality detection models. Our adaptive text-image fusion module is designed to learn optimal fusion rules between image and text features. Additionally, we introduce a word-level contrastive loss to guide the local-focused fusion of multimodal features and a sentence-level alignment loss to drive the global consistent fusion of multimodal features. Our paradigm is highly adaptable and can be easily embedded into existing image-based object detection models without any extra modification. We conduct extensive experiments on two multimodal detection datasets, and the results demonstrate that our LGF\(^2\) significantly improves their performance.

S. Miao and H. Zheng—contribute equally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/cocodataset/cocoapi.

References

  1. Cai, Y., et al.: Yolov4-5d: an effective and efficient object detector for autonomous driving. IEEE Trans. Instrum. Meas. 70, 1–13 (2021)

    Google Scholar 

  2. Chen, K., et al.: Mmdetection: Open mmlab detection toolbox and benchmark. arXiv (2019)

    Google Scholar 

  3. Chen, S., Li, Z., Tang, Z.: Relation R-CNN: a graph based relation-aware network for object detection. IEEE Signal Process. Lett. 27, 1680–1684 (2020)

    Article  Google Scholar 

  4. Dai, J., et al.: Deformable convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 764–773 (2017)

    Google Scholar 

  5. Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: Centernet: keypoint triplets for object detection. In: Proceedings of the IEEE/CVF International Conference On Computer Vision, pp. 6569–6578 (2019)

    Google Scholar 

  6. Fabbri, M., et al.: Motsynth: How can synthetic data help pedestrian detection and tracking? In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10849–10859 (2021)

    Google Scholar 

  7. Feng, C., et al.: Promptdet: Towards open-vocabulary detection using uncurated images. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IX. pp. 701–717. Springer (2022). https://doi.org/10.1007/978-3-031-20077-9_41

  8. Feng, D., et al.: Deep multi-modal object detection and semantic segmentation for autonomous driving: datasets, methods, and challenges. IEEE Trans. Intell. Transp. Syst. 22(3), 1341–1360 (2020)

    Article  Google Scholar 

  9. Ghiasi, G., Lin, T.Y., Le, Q.V.: Nas-fpn: Learning scalable feature pyramid architecture for object detection. In: CVPR, pp. 7036–7045 (2019)

    Google Scholar 

  10. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR, pp. 580–587 (2014)

    Google Scholar 

  11. Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 (2021)

  12. Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation networks for object detection. In: CVPR, pp. 3588–3597 (2018)

    Google Scholar 

  13. Jaeger, P.F., et al.: Retina u-net: Embarrassingly simple exploitation of segmentation supervision for medical object detection. In: Machine Learning for Health Workshop, pp. 171–183. PMLR (2020)

    Google Scholar 

  14. Kong, T., Sun, F., Liu, H., Jiang, Y., Li, L., Shi, J.: Foveabox: beyound anchor-based object detection. IEEE Trans. Image Process. 29, 7389–7398 (2020)

    Article  MATH  Google Scholar 

  15. Law, H., Deng, J.: Cornernet: Detecting objects as paired keypoints. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 734–750 (2018)

    Google Scholar 

  16. Li, H., Miao, S., Feng, R.: Dg-fpn: Learning dynamic feature fusion based on graph convolution network for object detection. In: ICME, pp. 1–6 (2020)

    Google Scholar 

  17. Li, L.H., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965–10975 (2022)

    Google Scholar 

  18. Li, Y., et al.: A deep learning-based hybrid framework for object detection and recognition in autonomous driving. IEEE Access 8, 194228–194239 (2020)

    Article  Google Scholar 

  19. Li, Y., Chen, Y., Wang, N., Zhang, Z.: Scale-aware trident networks for object detection. In: ICCV, pp. 6054–6063 (2019)

    Google Scholar 

  20. Li, Z., Dong, M., Wen, S., Hu, X., Zhou, P., Zeng, Z.: Clu-CNNs: object detection for medical images. Neurocomputing 350, 53–59 (2019)

    Article  Google Scholar 

  21. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR, pp. 2117–2125 (2017)

    Google Scholar 

  22. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV, pp. 2980–2988 (2017)

    Google Scholar 

  23. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  24. Liu, W., et al.: SSD: D. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  25. Liu, W., Liao, S., Ren, W., Hu, W., Yu, Y.: High-level semantic feature detection: a new perspective for pedestrian detection. In: CVPR, pp. 5187–5196 (2019)

    Google Scholar 

  26. Loey, M., Manogaran, G., Taha, M.H.N., Khalifa, N.E.M.: Fighting against Covid-19: a novel deep learning model based on yolo-v2 with Resnet-50 for medical face mask detection. Sustain. Urban Areas 65, 102600 (2021)

    Google Scholar 

  27. Miao, S., et al.: Balanced single-shot object detection using cross-context attention-guided network. Pattern Recogn. 122, 108258 (2022)

    Article  Google Scholar 

  28. Miao, S., Feng, R., Zhang, Y., Fan, W.: Learning class-based graph representation for object detection. In: AAAI, pp. 2752–2759 (2020)

    Google Scholar 

  29. Miao, S., Zheng, L., Jin, H., Feng, R.: Dynamically connected graph representation for object detection. In: International Conference on Neural Information Processing, pp. 347–358. Springer (2022). https://doi.org/10.1007/978-3-031-30111-7_30

  30. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)

    Google Scholar 

  31. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: CVPR (2016)

    Google Scholar 

  32. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. Arxiv (2015)

    Google Scholar 

  33. Singh, B., Davis, L.S.: An analysis of scale invariance in object detection snip. In: CVPR, pp. 3578–3587 (2018)

    Google Scholar 

  34. Tian, Z., Shen, C., Chen, H., He, T.: Fcos: Fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9627–9636 (2019)

    Google Scholar 

  35. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)

    Google Scholar 

  36. Xu, T., et al.: Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In: CVPR (2018)

    Google Scholar 

  37. Zhang, S., Chi, C., Yao, Y., Lei, Z., Li, S.Z.: Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In: CVPR (2020)

    Google Scholar 

  38. Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. Int. J. Comput. Vision 130(9), 2337–2348 (2022)

    Article  Google Scholar 

  39. Zhou, P., Ni, B., Geng, C., Hu, J., Xu, Y.: Scale-transferrable object detection. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 528–537 (2018)

    Google Scholar 

  40. Zhou, X., Zhuo, J., Krahenbuhl, P.: Bottom-up object detection by grouping extreme and center points. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 850–859 (2019)

    Google Scholar 

  41. Zhu, C., He, Y., Savvides, M.: Feature selective anchor-free module for single-shot object detection. In: Proceedings of the IEEE/CVF Conference On Computer Vision and Pattern Recognition, pp. 840–849 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Tiansuan Lab, Ant Group, Hangzhou, China

    Shuyu Miao, Lin Zheng & Hong Jin

  2. Business School, University of Shanghai for Science and Technology, Shanghai, China

    Hexiang Zheng

Authors
  1. Shuyu Miao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hexiang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hong Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shuyu Miao .

Editor information

Editors and Affiliations

  1. Democritus University of Thrace, Xanthi, Greece

    Lazaros Iliadis

  2. Democritus University of Thrace, Xanthi, Greece

    Antonios Papaleonidas

  3. Lancaster University, Lancaster, UK

    Plamen Angelov

  4. Teesside University, Middlesbrough, UK

    Chrisina Jayne

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Miao, S., Zheng, H., Zheng, L., Jin, H. (2023). LGF\(^2\): Local and Global Feature Fusion for Text-Guided Object Detection. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial Neural Networks and Machine Learning – ICANN 2023. ICANN 2023. Lecture Notes in Computer Science, vol 14260. Springer, Cham. https://doi.org/10.1007/978-3-031-44195-0_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-44195-0_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-44194-3

  • Online ISBN: 978-3-031-44195-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation