Abstract
Most of the existing visual reasoning networks primarily focus on aligning language with specific regions in images, but they often struggle to comprehend the complex spatial logical relationships within real-world scenes. Moreover, these networks lack interpretability. In order to solve the above problems, this paper proposes the Bilinear-MAC network with stronger reasoning ability. The MAC network is a single-stream network that performs reasoning tasks. It only uses the question representation as a control premise to extract image information. It can only understand simple images, but cannot understand complex images based on real scenes. The improved Bilinear-MAC network in this paper is a dual-stream network that performs reasoning tasks. It uses the problem representation as the control premise to extract image information, and uses the image representation as the control premise to extract problem information. As a result, it has a stronger understanding ability for real scene images with complex relationships and rich object types. The proposed network achieves an overall accuracy rate of 59.6% in the GQA dataset, which is 5.6% higher than that of the MAC network, and an accuracy rate of 99.1% in the CLEVR dataset. Experimental results demonstrate that the proposed network is capable of a stronger understanding of complex real-world visual scenes.
This work is supported by National Key R &D Program of China (Project No. 2022YFB4700400).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tan, H., Bansal, M.: LXMERT: learning cross-modality encoder representations from transformers. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China (2019)
Lu, J., Batra, D., Parikh, D., et al.: ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Adv. Neural Inf. Process. Syst. 32 (2019)
Kim, W., Son, B., Kim, I.: ViLT: vision-and-language transformer without convolution or region supervision. In: International Conference on Machine Learning, pp. 5583–5594. PMLR (2021)
Anderson, P., He, X., Buehler, C., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077–6086 (2018)
Lu, J., Yang, J., Batra, D., et al.: Hierarchical question-image co-attention for visual question answering. Adv. Neural Inf. Process. Syst. 29 (2016)
Yu, Z., Yu, J., Cui, Y., et al.: Deep modular co-attention networks for visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6281–6290 (2019)
Hudson, D.A., Manning, C.D.: Compositional attention networks for machine reasoning. In: International Conference on Learning Representations (2018)
Johnson, J., Hariharan, B., Van Der Maaten, L., et al.: CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901–2910 (2017)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)
Hudson, D.A., Manning, C.D.: GQA: a new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6700–6709 (2019)
Miao, Y., Cheng, W., He, S., et al.: Research on visual question answering based on GAT relational reasoning. Neural Process. Lett. 1–14 (2022)
Yu, Z., Yu, J., Cui, Y., et al.: Deep modular co-attention networks for visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6281–6290 (2019)
Rahman, T., Chou, S.H., Sigal, L., et al.: An improved attention for visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1653–1662 (2021)
Zhang, W., Yu, J., Zhao, W., et al.: DMRFNet: deep multimodal reasoning and fusion for visual question answering and explanation generation. Inf. Fusion 72, 70–79 (2021)
Qin, B., Hu, H., Zhuang, Y.: Deep residual weight-sharing attention network with low-rank attention for visual question answering. IEEE Trans. Multimed. 25, 4282–4295 (2023)
Li, J., Li, D., Savarese, S., et al.: Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zeng, J., Zheng, W., Lin, Y. (2024). The Bilinear-MAC Network for Visual Reasoning. In: Pan, L., Wang, Y., Lin, J. (eds) Bio-Inspired Computing: Theories and Applications. BIC-TA 2023. Communications in Computer and Information Science, vol 2062. Springer, Singapore. https://doi.org/10.1007/978-981-97-2275-4_6
Download citation
DOI: https://doi.org/10.1007/978-981-97-2275-4_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-2274-7
Online ISBN: 978-981-97-2275-4
eBook Packages: Computer ScienceComputer Science (R0)