Skip to main content
SpringerLink
Log in

Lightweight Image Captioning Model Based on Knowledge Distillation

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14555))

Included in the following conference series:

  • 316 Accesses

Abstract

The performance of image captioning models based on deep learning has been significantly improved compared with traditional algorithms. However, due to the complex network structure and huge parameters, these image captioning models are hard to apply to resource-constrained appliances such as mobile embedded systems. To address this issue, we propose two lightweight image captioning models based on two pre-training models, i.e. \({M^2}\) Transformer and CaMEL. To improve the performances of the proposed lightweight models, we also develop an optimization training strategy based on knowledge distillation. Specifically, we design knowledge distillation loss functions in the encoder, decoder, and response modules, which can use the multi-modal feature mapping of the original models to transfer prior knowledge, improving the feature learning ability and generalization ability of the lightweight models. Experimental results show that the proposed lightweight models are superior to the original pre-trained models in performance when reducing the parameters greatly. Besides, the proposed lightweight models also have a good knowledge transfer ability.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Anderson, P., Fernando, B., Johnson, M., Gould, S.: SPICE: semantic propositional image caption evaluation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 382–398. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_24

    Chapter  Google Scholar 

  2. Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077–6086 (2018)

    Google Scholar 

  3. Atliha, V., Šešok, D.: Image-captioning model compression. Appl. Sci. 12(3), 1638 (2022)

    Article  Google Scholar 

  4. Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72 (2005)

    Google Scholar 

  5. Barraco, M., Stefanini, M., Cornia, M., Cascianelli, S., Baraldi, L., Cucchiara, R.: CaMEL: mean teacher learning for image captioning. In: 2022 26th International Conference on Pattern Recognition (ICPR), pp. 4087–4094. IEEE (2022)

    Google Scholar 

  6. Cornia, M., Baraldi, L., Cucchiara, R.: SMArT: training shallow memory-aware transformers for robotic explainability. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1128–1134. IEEE (2020)

    Google Scholar 

  7. Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R.: Meshed-memory transformer for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10578–10587 (2020)

    Google Scholar 

  8. Denil, M., Shakibi, B., Dinh, L., Ranzato, M., De Freitas, N.: Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems, vol. 26 (2013)

    Google Scholar 

  9. Dong, J., Hu, Z., Zhou, Y.: Revisiting knowledge distillation for image captioning. In: Fang, L., Chen, Y., Zhai, G., Wang, J., Wang, R., Dong, W. (eds.) Artificial Intelligence CICAI 2021. Lecture Notes in Computer Science, vol. 13069, pp. 613–625. Springer, Cham (2021)

    Google Scholar 

  10. Hahn, S., Choi, H.: Self-knowledge distillation in natural language processing. arXiv preprint arXiv:1908.01851 (2019)

  11. Herdade, S., Kappeler, A., Boakye, K., Soares, J.: Image captioning: transforming objects into words. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  12. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  13. Hsieh, H.Y., Huang, S.A., Leu, J.S.: Implementing a real-time image captioning service for scene identification using embedded system. Multimed. Tools Appl. 80, 12525–12537 (2021)

    Article  Google Scholar 

  14. Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4634–4643 (2019)

    Google Scholar 

  15. Huang, Y., Chen, J.: Teacher-critical training strategies for image captioning. arXiv preprint arXiv:2009.14405 (2020)

  16. Jiao, X., et al.: TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351 (2019)

  17. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137 (2015)

    Google Scholar 

  18. Li, X., Guo, D., Liu, H., Sun, F.: Robotic indoor scene captioning from streaming video. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6109–6115. IEEE (2021)

    Google Scholar 

  19. Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out, pp. 74–81 (2004)

    Google Scholar 

  20. Luo, R.: A better variant of self-critical sequence training. arXiv preprint arXiv:2003.09971 (2020)

  21. Luo, Y., et al.: Dual-level collaborative transformer for image captioning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 2286–2293 (2021)

    Google Scholar 

  22. Ma, Y., Ji, J., Sun, X., Zhou, Y., Ji, R.: Towards local visual modeling for image captioning. Pattern Recogn. 138, 109420 (2023)

    Article  Google Scholar 

  23. Pan, Y., Yao, T., Li, Y., Mei, T.: X-linear attention networks for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10971–10980 (2020)

    Google Scholar 

  24. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)

    Google Scholar 

  25. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)

    Google Scholar 

  26. Sun, S., Cheng, Y., Gan, Z., Liu, J.: Patient knowledge distillation for BERT model compression. arXiv preprint arXiv:1908.09355 (2019)

  27. Tan, J.H., Chan, C.S., Chuah, J.H.: COMIC: toward a compact image captioning model with attention. IEEE Trans. Multimed. 21(10), 2686–2696 (2019)

    Article  Google Scholar 

  28. Tan, J.H., Chan, C.S., Chuah, J.H.: End-to-end supermask pruning: learning to prune image captioning models. Pattern Recogn. 122, 108366 (2022)

    Article  Google Scholar 

  29. Tan, J.H., Tan, Y.H., Chan, C.S., Chuah, J.H.: ACORT: a compact object relation transformer for parameter efficient image captioning. Neurocomputing 482, 60–72 (2022)

    Article  Google Scholar 

  30. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  31. Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)

    Google Scholar 

  32. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In: Advances in Neural Information Processing Systems, vol. 33, pp. 5776–5788 (2020)

    Google Scholar 

  33. Yang, X., Zhang, H., Cai, J.: Auto-encoding and distilling scene graphs for image captioning. IEEE Trans. Pattern Anal. Mach. Intell. 44(5), 2313–2327 (2020)

    Google Scholar 

  34. Zhang, Q., Cheng, X., Chen, Y., Rao, Z.: Quantifying the knowledge in a DNN to explain knowledge distillation for classification. IEEE Trans. Pattern Anal. Mach. Intell. 45(4), 5099–5113 (2022)

    Google Scholar 

  35. Zhang, X., et al.: RSTNet: captioning with adaptive attention on visual and non-visual words. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15465–15474 (2021)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part by the Natural Science Foundation of Guangxi, China (No. 2021GXNSFAA220058).

Author information

Authors and Affiliations

  1. School of Computer and Electronics Information, Guangxi University, Nanning, 530004, China

    Zhenlei Cui, Zhenhua Tang, Jianze Li & Kai Chen

  2. Guangxi Key Laboratory of Multimedia Communication and Network Technology, Nanning, 530004, China

    Zhenhua Tang

Authors
  1. Zhenlei Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenhua Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianze Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenhua Tang .

Editor information

Editors and Affiliations

  1. University of Amsterdam, Amsterdam, The Netherlands

    Stevan Rudinac

  2. Delft University of Technology, Delft, The Netherlands

    Alan Hanjalic

  3. Delft University of Technology, Delft, The Netherlands

    Cynthia Liem

  4. University of Amsterdam, Amsterdam, The Netherlands

    Marcel Worring

  5. Reykjavik University, Reykjavik, Iceland

    Björn Þór Jónsson

  6. Microsoft Research Lab – Asia, Beijing, China

    Bei Liu

  7. The University of Tokyo, Tokyo, Japan

    Yoko Yamakata

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cui, Z., Tang, Z., Li, J., Chen, K. (2024). Lightweight Image Captioning Model Based on Knowledge Distillation. In: Rudinac, S., et al. MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14555. Springer, Cham. https://doi.org/10.1007/978-3-031-53308-2_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-53308-2_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-53307-5

  • Online ISBN: 978-3-031-53308-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation