Abstract
Research in Document Intelligence and especially in Document Key Information Extraction (DocKIE) has been mainly solved as Token Classification problem. Recent breakthroughs in both natural language processing (NLP) and computer vision helped building document-focused pre-training methods, leveraging a multimodal understanding of the document text, layout and image modalities.
However, these breakthroughs also led to the emergence of a new DocKIE subtask of extractive document Question Answering (DocQA), as part of the Machine Reading Comprehension (MRC) research field.
In this work, we compare the Question Answering approach with the classical token classification approach for document key information extraction. We designed experiments to benchmark five different experimental setups : raw performances, robustness to noisy environment, capacity to extract long entities, fine-tuning speed on Few-Shot Learning and finally Zero-Shot Learning.
Our research showed that when dealing with clean and relatively short entities, it is still best to use token classification-based approach, while the QA approach could be a good alternative for noisy environment or long entities use-cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (2019). http://aclanthology.org/N19-1423.pdf
Douzon, T., Duffner, S., Garcia, C., Espinas, J.: Improving information extraction on business documents with specific pre-training tasks. In: Document Analysis Systems 15th IAPR International Workshop, DAS 2022. LNCS, vol. 13237, pp. 111–125. Springer International Publishing, La Rochelle, France (2022). https://doi.org/10.1007/978-3-031-06555-2_8, http://hal.archives-ouvertes.fr/hal-03676134
Hendrycks, D., Burns, C., Chen, A., Ball, S.: CUAD: an expert-annotated NLP dataset for legal contract review. arXiv preprint arXiv:2103.06268 (2021)
Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: Layoutlmv3: pre-training for document AI with unified text and image masking. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 4083–4091. MM 2022, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3503161.3548112
Huang, Z., et al.: ICDAR 2019 competition on scanned receipt OCR and information extraction, pp. 1516–1520 (2019). http://arxiv.org/pdf/2103.10213.pdf
Jaume, G., Kemal Ekenel, H., Thiran, J.P.: FUNSD: a dataset for form understanding in noisy scanned documents. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), vol. 2, pp. 1–6 (2019). https://doi.org/10.1109/ICDARW.2019.10029
Kwiatkowski, T., et al.: Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguist. 7, 452–466 (2019). https://doi.org/10.1162/tacl_a_00276, http://aclanthology.org/Q19-1026
Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., Li, J.: A unified MRC framework for named entity recognition (2019). https://doi.org/10.48550/ARXIV.1910.11476, http://arxiv.org/abs/1910.11476
Mengge, X., Yu, B., Zhang, Z., Liu, T., Zhang, Y., Wang, B.: Coarse-to-fine pre-training for named entity recognition. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6345–6354. Association for Computational Linguistics, November 2020. https://doi.org/10.18653/v1/2020.emnlp-main.514, http://aclanthology.org/2020.emnlp-main.514
Oussaid, I., Vanhuffel, W., Ratnamogan, P., Hajaiej, M., Mathey, A., Gilles, T.: Information extraction from visually rich documents with font style embeddings. CoRR abs/2111.04045 (2021). http://arxiv.org/abs/2111.04045
Park, S., et al.: Cord: a consolidated receipt dataset for post-OCR parsing (2019)
Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: Unanswerable questions for SQuAD. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789. Association for Computational Linguistics, Melbourne, Australia, July 2018. https://doi.org/10.18653/v1/P18-2124, http://aclanthology.org/P18-2124
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392. Association for Computational Linguistics, Austin, Texas, November 2016. https://doi.org/10.18653/v1/D16-1264, http://aclanthology.org/D16-1264
Ramshaw, L.A., Marcus, M.P.: Text chunking using transformation-based learning (1995). https://doi.org/10.48550/ARXIV.CMP-LG/9505040, http://arxiv.org/abs/cmp-lg/9505040
Saha, A., Finegan-Dollak, C., Verma, A.: Position masking for improved layout-aware document understanding. CoRR abs/2109.00442 (2021). http://arxiv.org/abs/2109.00442
Shrimal, A., Jain, A., Mehta, K., Yenigalla, P.: NER-MQMRC: formulating named entity recognition as multi question machine reading comprehension. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pp. 230–238. Association for Computational Linguistics, Hybrid: Seattle, Washington + Online, July 2022. https://doi.org/10.18653/v1/2022.naacl-industry.26
Stanislawek, T., et al.: Kleister: key information extraction datasets involving long documents with complex layouts. CoRR abs/2105.05796 (2021). http://arxiv.org/abs/2105.05796
Vu, H.M., Nguyen, D.T.N.: Revising FUNSD dataset for key-value detection in document images. arXiv preprint arXiv:2010.05322 (2020)
Vu, H.M., Nguyen, D.T.: Revising FUNSD dataset for key-value detection in document images. CoRR abs/2010.05322 (2020). http://arxiv.org/abs/2010.05322
Wang, Q., et al.: Learning to extract attribute value from product via question answering: a multi-task approach. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 47–55. KDD 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3394486.3403047
Wang, W.C., Mueller, J.: Detecting label errors in token classification data (2022). https://doi.org/10.48550/ARXIV.2210.03920, http://arxiv.org/abs/2210.03920
Xu, Y., et al.: LayoutLMv2: multi-modal pre-training for visually-rich document understanding. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2579–2591. Association for Computational Linguistics, August 2021. https://doi.org/10.18653/v1/2021.acl-long.201
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: LayoutLM: pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1192–1200. KDD 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3394486.3403172
Zaheer, M., et al.: Big bird: transformers for longer sequences. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 17283–17297. Curran Associates, Inc. (2020). www.proceedings.neurips.cc/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf
Zaheer, M., et al.: Big bird: transformers for longer sequences. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 17283–17297. Curran Associates, Inc. (2020). www.proceedings.neurips.cc/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf
Zeng, C., Li, S., Li, Q., Hu, J., Hu, J.: A survey on machine reading comprehension: tasks, evaluation metrics and benchmark datasets (2020). https://doi.org/10.48550/ARXIV.2006.11880, http://arxiv.org/abs/2006.11880
Zhou, W., Chen, M.: Learning from noisy labels for entity-centric information extraction. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5381–5392. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, November 2021. https://doi.org/10.18653/v1/2021.emnlp-main.437
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lam, L., Ratnamogan, P., Tang, J., Vanhuffel, W., Caspani, F. (2023). Information Extraction from Documents: Question Answering Vs Token Classification in Real-World Setups. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14188. Springer, Cham. https://doi.org/10.1007/978-3-031-41679-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-41679-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41678-1
Online ISBN: 978-3-031-41679-8
eBook Packages: Computer ScienceComputer Science (R0)
