Skip to main content

A Scheduled Mask Method for TextVQA

  • Conference paper
  • First Online:
Cognitive Computing – ICCC 2022 (ICCC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13734))

Included in the following conference series:

  • 255 Accesses

Abstract

At present, many successful applications use deep learning method in the field of Visual Question Answering (VQA). With the introduction of Optical Character Recognition (OCR), Text-based Visual Question Answering (TextVQA) tasks have a mature basic structure, a transformer-based iterative decoding prediction module. However, there is a problem in the current models: the training process of the model is inconsistent with the inference process. This inconsistency is shown in the different input and the different iteration prediction steps. We propose a scheduled mask method. After using this method, our model can gradually adapt to the situation without the ground truth answer input in the training process. We have verified the effectiveness of our method on the TextVQA dataset and exceeded the performance of other models previously proposed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Google Scholar 

  2. Zhao, Z.Q., Zheng, P., Xu, S.t., Wu, X.: Object detection with deep learning: a review. IEEE Trans. Neural Networks Learn. Syst. 30(11), 3212–3232 (2019)

    Google Scholar 

  3. Amit, Y., Felzenszwalb, P., Girshick, R.: Object detection. Computer Vision : A Reference Guide, pp. 1–9 (2020)

    Google Scholar 

  4. Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., Cottrell, G.: Understanding convolution for semantic segmentation. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1451–1460. IEEE (2018)

    Google Scholar 

  5. Guo, Y., Liu, Y., Georgiou, T., Lew, M.S.: A review of semantic segmentation using deep neural networks. Int. J. Multimed. Inf. Retrieval 7(2), 87–93 (2018)

    Article  Google Scholar 

  6. Gregor, K., Danihelka, I., Graves, A., Rezende, D., Wierstra, D.: Draw: a recurrent neural network for image generation. In: International Conference on Machine Learning, PMLR, pp. 1462–1471 (2015)

    Google Scholar 

  7. Qiao, T., Zhang, J., Xu, D., Tao, D.: Mirrorgan: learning text-to-image generation by redescription. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1505–1514 (2019)

    Google Scholar 

  8. Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., Brown, D.: Text classification algorithms: a survey. Information 10(4), 150 (2019)

    Article  Google Scholar 

  9. Mirończuk, M.M., Protasiewicz, J.: A recent overview of the state-of-the-art elements of text classification. Expert Syst. Appl. 106, 36–54 (2018)

    Article  Google Scholar 

  10. Piskorski, J., Yangarber, R.: Information extraction: past, present and future. In: Multi-source, Multilingual Information Extraction and Summarization, pp. 23–49. Springer (2013)

    Google Scholar 

  11. Olivetti, E.A., Cole, J.M., Kim, E., Kononova, O., Ceder, G., Han, T.Y.J., Hiszpanski, A.M.: Data-driven materials research enabled by natural language processing and information extraction. Appl. Phys. Rev. 7(4), 041317 (2020)

    Article  Google Scholar 

  12. Haque, S., LeClair, A., Wu, L., McMillan, C.: Improved automatic summarization of subroutines via attention to file context. In: Proceedings of the 17th International Conference on Mining Software Repositories, pp. 300–310 (2020)

    Google Scholar 

  13. Moreno, L., Marcus, A.: Automatic software summarization: the state of the art. In: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, pp. 530–531 (2018)

    Google Scholar 

  14. Hu, R., Singh, A., Darrell, T., Rohrbach, M.: Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9992–10002 (2020)

    Google Scholar 

  15. Singh, A., et al.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8317–8326 (2019)

    Google Scholar 

  16. Gao, C., et al.: Structured multimodal attentions for textvqa. IEEE Trans. Pattern Anal. Mach. Intell. (2021)

    Google Scholar 

  17. Li, X., Wu, B., Song, J., Gao, L., Zeng, P., Gan, C.: Text-instance graph: exploring the relational semantics for text-based visual question answering. Pattern Recogn. 124, 108455 (2022)

    Article  Google Scholar 

  18. Liu, F., Xu, G., Wu, Q., Du, Q., Jia, W., Tan, M.: Cascade reasoning network for text-based visual question answering. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 4060–4069 (2020)

    Google Scholar 

  19. Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: Interspeech. Volume 2, Makuhari, pp. 1045–1048 (2010)

    Google Scholar 

  20. Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N.: Scheduled sampling for sequence prediction with recurrent neural networks. In: Advances in Neural Information Processing Systems 28 (2015)

    Google Scholar 

  21. Yu, L., Zhang, W., Wang, J., Yu, Y.: Seqgan: sequence generative adversarial nets with policy gradient. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  22. Finn, C., Goodfellow, I., Levine, S.: Unsupervised learning for physical interaction through video prediction. In: Advances in Neural Information Processing Systems 29 (2016)

    Google Scholar 

  23. Martinez, J., Black, M.J., Romero, J.: On human motion prediction using recurrent neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2891–2900 (2017)

    Google Scholar 

  24. Ranzato, M., Chopra, S., Auli, M., Zaremba, W.: Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 (2015)

  25. Lamb, A.M., Alias Parth Goyal, A.G., Zhang, Y., Zhang, S., Courville, A.C., Bengio, Y.: Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems 29 (2016)

    Google Scholar 

  26. Schmidt, F.: Generalization in generation: A closer look at exposure bias. arXiv preprint arXiv:1910.00292 (2019)

  27. Krasin, I., et al.: Openimages: a public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github.com/openimages 2(3) (2017) 18

  28. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433 (2015)

    Google Scholar 

  29. Borisyuk, F., Gordo, A., Sivakumar, V.: Rosetta: large scale system for text detection and recognition in images. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 71–79 (2018)

    Google Scholar 

  30. Zeng, G., Zhang, Y., Zhou, Y., Yang, X.: Beyond ocr+ vqa: involving ocr into the flow for robust and accurate textvqa. In: Proceedings of the 29th ACM International Conference on Multimedia (2021) 376–385

    Google Scholar 

  31. Zhu, Q., Gao, C., Wang, P., Wu, Q.: Simple is not easy: A simple strong baseline for textvqa and textcaps. arXiv preprint arXiv:2012.05153 2 (2020)

  32. Han, W., Huang, H., Han, T.: Finding the evidence: Localization-aware answer prediction for text visual question answering. arXiv preprint arXiv:2010.02582 (2020)

Download references

Acknowledgment

This work was supported by National Science Foundation of China (No. 61862021) and Hainan Provincial Natural Science Foundation of China (No. 620RC565).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ting Jin .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Han, M., Jin, T., Lin, W. (2022). A Scheduled Mask Method for TextVQA. In: Yang, Y., Wang, X., Zhang, LJ. (eds) Cognitive Computing – ICCC 2022. ICCC 2022. Lecture Notes in Computer Science, vol 13734. Springer, Cham. https://doi.org/10.1007/978-3-031-23585-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23585-6_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23584-9

  • Online ISBN: 978-3-031-23585-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics