Abstract
Video captioning aims to describe the main content of a given video in natural language, which has become a research hotspot because of its wide potential application prospect. Semantic information, as a priori knowledge, is often applied to improve the caption quality, but the scope of these semantic information is relatively small, resulting in insufficient coverage of video attributes. In this paper, we introduce external knowledge from ConceptNet to expand the semantic coverage, so that the model can refer to more semantic information. In addition, a multi-feature fusion is proposed to obtain more informative video features and higher quality semantic features. Experimental results on the MSVD and MSRVTT datasets show that the proposed method can greatly improve the caption diversity and model performance, surpass all previous models in all evaluation metrics, and achieve the new state-of-the-art results.
J.-W. Miao and H. Shao—These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aafaq, N., Akhtar, N., Liu, W., Gilani, S.Z., Mian, A.: Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12487–12496 (2019)
Chen, H., Lin, K., Maye, A., Li, J., Hu, X.: A semantics-assisted video captioning model trained with scheduled sampling. Front. Robot. AI 7, 475767 (2020)
Fang, H., et al.: From captions to visual concepts and back. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1473–1482 (2015)
Guadarrama, S., et al.: YouTube2Text: recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2712–2719 (2013)
Liu, H., Singh, P.: ConceptNet-a practical commonsense reasoning tool-kit. BT Technol. J. 22(4), 211–226 (2004). https://doi.org/10.1023/B:BTTJ.0000047600.45421.6d
Liu, S., Ren, Z., Yuan, J.: SibNet: sibling convolutional encoder for video captioning. In: Proceedings of the 2018 ACM on Multimedia Conference, pp. 1425–1434 (2018)
Pan, B., et al.: Spatio-temporal graph for video captioning with knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10870–10879 (2020)
Pei, W., Zhang, J., Wang, X., Ke, L., Shen, X., Tai, Y.W.: Memory-attended recurrent network for video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8347–8356 (2019)
Wang, B., Ma, L., Zhang, W., Jiang, W., Wang, J., Liu, W.: Controllable video captioning with pos sequence guidance based on gated fusion network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 2641–2650 (2019)
Xu, J., Mei, T., Yao, T., Rui, Y.: MSR-VTT: a large video description dataset for bridging video and language. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5288–5296 (2016)
Zhang, Z., et al.: Object relational graph with teacher-recommended learning for video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13278–13288 (2020)
Acknowledgment
Supported by National Natural Science Foundation of China Nos. 61972059, 61773272, 61602332; Natural Science Foundation of the Jiangsu Higher Education Institutions of China No. 19KJA230001, Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University No. 93K172016K08; Postgraduate Research & Practice Innovation Program of Jiangsu Province SJCX20_1063; Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Miao, JW., Shao, H., Ji, Y., Li, Y., Liu, CP. (2021). Video Captioning with External Knowledge Assistance and Multi-feature Fusion. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Communications in Computer and Information Science, vol 1517. Springer, Cham. https://doi.org/10.1007/978-3-030-92310-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-92310-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92309-9
Online ISBN: 978-3-030-92310-5
eBook Packages: Computer ScienceComputer Science (R0)