Abstract
Concatenation of text (question-answer) and image has been the bedrock of most visual language systems. Existing models concatenate the text (question-answer) and image inputs in a forced manner. In this paper, we introduce a region descriptive pre-training approach with self-attention towards VQA. The model is a new learning method that uses the image region descriptions combined with object labels to create a proper alignment between the text(question-answer) and the image inputs. We study the text associated with each image and discover that extracting the region descriptions from the image and using it during training greatly improves the model’s performance. In this research work, we use the region description extracted from the images as a bridge to map the text and image inputs. The addition of region description makes our model perform better against some recent state-of-the-art models. Experiments demonstrated in this paper show that our model significantly outperforms most of these models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018)
Li, X., et al.: Oscar: object-semantics aligned pre-training for vision-language tasks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12375, pp. 121–137. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58577-8_8
Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265
Li, L., Yatskar, M., Yin, D., Hsieh, C., Chang, K.: VISUALBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557
Agrawal, A., et al.: VQA: visual question answering. arXiv:1906.08237
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.: XLNet: generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 (2020)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding, arXiv preprint (2019)
Tan, H., Bansal, M.: LXMERT.: learning cross-modality encoder representations from transformers. In: EMNLP (2019)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497 (2016)
Lin, T., et al.: Microsoft COCO: common objects in context. arXiv preprint arXiv:1405.0312
Kim, J.-H., Jun, J., Zhang, B.-T.: Bilinear attention networks. In: NIPS (2018)
Gao, P., et al.: Dynamic fusion with intra- and inter-modality attention flow for visual question answering. arXiv preprint arXiv:1812.05252 (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
Vaswani, A., et al.: Attention Is All You Need. arXiv preprint: 1706.03762 (2017)
Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. arXiv preprint (2016)
Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Annual Meeting of the Association for Computational Linguistics (2018)
Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguist. 2, 67–78 (2014)
Akbari, H., et al.: VATT: transformer for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104.11178 (2021)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Su, W., et al.: VL-BERT: pre-training of generic visual-linguistic representations. In: ICLR (2020)
Yen-Chun, C., et al.: UNITER: UNiversal image-TExt representation learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12375, pp. 104–120. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58577-8_7
Acknowledgment
This research was partly supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program (IITP-2021-2020-0-01808) supervised by the IITP (Institute of Information & Communications Technology Planning & Evaluation)[50%] and the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2021R1A2C3011169)[50%].
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kolawole, B.B., Lee, M. (2021). A Region Descriptive Pre-training Approach with Self-attention Towards Visual Question Answering. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Communications in Computer and Information Science, vol 1517. Springer, Cham. https://doi.org/10.1007/978-3-030-92310-5_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-92310-5_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92309-9
Online ISBN: 978-3-030-92310-5
eBook Packages: Computer ScienceComputer Science (R0)