Skip to main content
Springer Nature Link
Log in

A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documents

  • Conference paper
  • First Online:
Document Analysis and Recognition - ICDAR 2024 (ICDAR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14807))

Included in the following conference series:

  • 537 Accesses

Abstract

Document Understanding is an evolving field in Natural Language Processing (NLP). In particular, visual and spatial features are essential in addition to the raw text itself and hence, several multimodal models were developed in the field of Visual Document Understanding (VDU). However, while research is mainly focused on Key Information Extraction (KIE), Relation Extraction (RE) between identified entities is still under-studied. For instance, RE is crucial to regroup entities or obtain a comprehensive hierarchy of data in a document. In this paper, we present a model that, initialized from LayoutLMv3, can match or outperform the current state-of-the-art results in RE applied to Visually-Rich Documents (VRD) on FUNSD and CORD datasets, without any specific pre-training and with fewer parameters. We also report an extensive ablation study performed on FUNSD, highlighting the great impact of certain features and modelization choices on the performances.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Baldini Soares, L., FitzGerald, N., Ling, J., Kwiatkowski, T.: Matching the blanks: distributional similarity for relation learning. In: Korhonen, A., Traum, D., Màrquez, L. (eds.) Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2895–2905. Association for Computational Linguistics, Florence, Italy, July 2019. https://doi.org/10.18653/v1/P19-1279, https://aclanthology.org/P19-1279

  2. Brill, E.: A simple rule-based part of speech tagger. In: Third Conference on Applied Natural Language Processing, pp. 152–155. Association for Computational Linguistics, Trento, Italy, March 1992. https://doi.org/10.3115/974499.974526, https://aclanthology.org/A92-1021

  3. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2019)

    Google Scholar 

  4. Fang, Z., Cao, Y., Li, Q., Zhang, D., Zhang, Z., Liu, Y.: Joint entity linking with deep reinforcement learning. In: The World Wide Web Conference, WWW 2019, pp. 438–447. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3308558.3313517

  5. Fang, Z., Cao, Y., Li, R., Zhang, Z., Liu, Y., Wang, S.: High quality candidate generation and sequential graph attention network for entity linking. In: Proceedings of The Web Conference 2020, WWW 2020, pp. 640–650. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3366423.3380146

  6. Hong, T., Kim, D., Ji, M., Hwang, W., Nam, D., Park, S.: BROS: a pre-trained language model focusing on text and layout for better key information extraction from documents (2022)

    Google Scholar 

  7. Huang, Y.Y., Wang, W.Y.: Deep residual learning for weakly-supervised relation extraction. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1803–1807. Association for Computational Linguistics, Copenhagen, Denmark, September 2017. https://doi.org/10.18653/v1/D17-1191, https://aclanthology.org/D17-1191

  8. Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: LayoutLMv3: pre-training for document AI with unified text and image masking. In: Proceedings of the 30th ACM International Conference on Multimedia, MM 2022, pp. 4083–4091. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3503161.3548112

  9. Jaume, G., Ekenel, H.K., Thiran, J.P.: FUNSD: a dataset for form understanding in noisy scanned documents. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), vol. 2, pp. 1–6. IEEE (2019)

    Google Scholar 

  10. Li, Y., et al.: StrucTexT: structured text understanding with multi-modal transformers. In: Proceedings of the 29th ACM International Conference on Multimedia, MM 2021, pp. 1912–1920. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3474085.3475345

  11. Luo, C., Cheng, C., Zheng, Q., Yao, C.: GeoLayoutLM: geometric pre-training for visual information extraction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7092–7101, June 2023

    Google Scholar 

  12. Ma, Y., Wang, A., Okazaki, N.: DREEAM: guiding attention with evidence for improving document-level relation extraction (2023)

    Google Scholar 

  13. Park, S., et al.: CORD: a consolidated receipt dataset for Post-OCR parsing. In: Workshop on Document Intelligence at NeurIPS 2019 (2019)

    Google Scholar 

  14. Ramshaw, L.A., Marcus, M.P.: Text chunking using transformation-based learning (1995)

    Google Scholar 

  15. Vu, H.M., Nguyen, D.T.N.: Revising FUNSD dataset for key-value detection in document images. arXiv preprint arXiv:2010.05322 (2020)

  16. Wang, C., Liu, X., Chen, Z., Hong, H., Tang, J., Song, D.: DeepStruct: pretraining of language models for structure prediction. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Findings of the Association for Computational Linguistics: ACL 2022, pp. 803–823. Association for Computational Linguistics, Dublin, Ireland, May 2022. https://doi.org/10.18653/v1/2022.findings-acl.67, https://aclanthology.org/2022.findings-acl.67

  17. Wang, R., et al.: K-Adapter: infusing knowledge into pre-trained models with adapters. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1405–1418. Association for Computational Linguistics, Online, August 2021. https://doi.org/10.18653/v1/2021.findings-acl.121, https://aclanthology.org/2021.findings-acl.121

  18. Wang, Z., Zhan, M., Liu, X., Liang, D.: DocStruct: a multimodal method to extract hierarchy structure in document for general form understanding. In: Cohn, T., He, Y., Liu, Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 898–908. Association for Computational Linguistics, Online, November 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.80, https://aclanthology.org/2020.findings-emnlp.80

  19. Xu, Y., et al.: LayoutLMv2: multi-modal pre-training for visually-rich document understanding (2022)

    Google Scholar 

  20. Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: LayoutLM: pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2020. ACM, August 2020. https://doi.org/10.1145/3394486.3403172

  21. Zhang, Y., Bo, Z., Wang, R., Cao, J., Li, C., Bao, Z.: Entity relation extraction as dependency parsing in visually rich documents. In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.T. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2759–2768. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, November 2021. https://doi.org/10.18653/v1/2021.emnlp-main.218, https://aclanthology.org/2021.emnlp-main.218

  22. Zhang, Y., Zhong, V., Chen, D., Angeli, G., Manning, C.D.: Position-aware attention and supervised data improve slot filling. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pp. 35–45 (2017). https://nlp.stanford.edu/pubs/zhang2017tacred.pdf

  23. Zhang, Y., Zhong, V., Chen, D., Angeli, G., Manning, C.D.: Position-aware attention and supervised data improve slot filling. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 35–45. Association for Computational Linguistics, Copenhagen, Denmark, September 2017. https://doi.org/10.18653/v1/D17-1004, https://aclanthology.org/D17-1004

  24. Zhou, W., Chen, M.: An improved baseline for sentence-level relation extraction. In: He, Y., Ji, H., Li, S., Liu, Y., Chang, C.H. (eds.) Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 161–168. Association for Computational Linguistics, Online Only, November 2022. https://aclanthology.org/2022.aacl-short.21

  25. Zhou, W., Huang, K., Ma, T., Huang, J.: Document-level relation extraction with adaptive thresholding and localized context pooling. In: Proceedings of the AAAI Conference on Artificial Intelligence (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Ecole Polytechnique, Palaiseau, France

    Wiam Adnan

  2. BNP Paribas CIB, Paris, France

    Wiam Adnan, Joel Tang, Yassine Bel Khayat Zouggari, Seif Edinne Laatiri, Laurent Lam & Fabien Caspani

Authors
  1. Wiam Adnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joel Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yassine Bel Khayat Zouggari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seif Edinne Laatiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Laurent Lam
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fabien Caspani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wiam Adnan .

Editor information

Editors and Affiliations

  1. Luleå Tekniska Universitet, Luleå, Sweden

    Elisa H. Barney Smith

  2. Luleå Tekniska Universitet, Luleå, Sweden

    Marcus Liwicki

  3. Tsinghua University, Beijing, China

    Liangrui Peng

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Adnan, W., Tang, J., Zouggari, Y.B.K., Laatiri, S.E., Lam, L., Caspani, F. (2024). A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documents. In: Barney Smith, E.H., Liwicki, M., Peng, L. (eds) Document Analysis and Recognition - ICDAR 2024. ICDAR 2024. Lecture Notes in Computer Science, vol 14807. Springer, Cham. https://doi.org/10.1007/978-3-031-70546-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-70546-5_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-70545-8

  • Online ISBN: 978-3-031-70546-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation