Abstract
Key value pair (KVP) extraction or Named Entity Recognition (NER) from visually rich documents has been an active area of research in document understanding and data extraction domain. Several transformer based models such as LayoutLMv2 [1], LayoutLMv3 [2], and LiLT [3] have emerged achieving state of the art results. However, addition of even a single new class to the existing model requires (a) re-annotation of entire training dataset to include this new class and (b) retraining the model again. Both of these issues really slow down the deployment of updated model.
We present ProtoNER: Prototypical Network based end-to-end KVP extraction model that allows addition of new classes to an existing model while requiring minimal number of newly annotated training samples. The key contributions of our model are: (1) No dependency on dataset used for initial training of the model, which alleviates the need to retain original training dataset for longer duration as well as data re-annotation which is very time consuming task, (2) No intermediate synthetic data generation which tends to add noise and results in model’s performance degradation, and (3) Hybrid loss function which allows model to retain knowledge about older classes as well as learn about newly added classes.
Experimental results show that ProtoNER finetuned with just 30 samples is able to achieve similar results for the newly added classes as that of regular model finetuned with 2600 samples.
Work done while author was working at IBM Research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The addition of multiple classes sequentially (one at a time) vs. all at same time results in similar accuracy.
References
Xu, Y., et al.: LayoutLMv2: multi-modal pre-training for visually-rich document understanding. In: ACL (2021)
Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: LayoutLMv3: pre-training for document AI with unified text and image masking. In: Proceedings of the 30th ACM International Conference on Multimedia (2022)
Wang, J., Jin, L., Ding, K.: LiLT: a simple yet effective language-independent layout transformer for structured document understanding. arXiv preprint arXiv:2202.13669 (2022)
Lee, C.Y., et al.: FormNet: structural encoding beyond sequential modeling in form document information extraction. arXiv preprint arXiv:2203.08411 (2022)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Liu, X., Gao, F., Zhang, Q., Zhao, H.: Graph convolution for multimodal information extraction from visually rich documents. arXiv preprint arXiv:1903.11279 (2019)
Watanabe, T., Luo, Q., Sugie, N.: Layout recognition of multi-kinds of table-form documents. IEEE Trans. Pattern Anal. Mach. Intell. 17(4), 432–445 (1995)
Seki, M., Fujio, M., Nagasaki, T., Shinjo, H., Marukawa, K.: Information management system using structure analysis of paper/electronic documents and its application. In: Proceedings of International Conference on Document Analysis and Recognition (ICDAR), pp. 689–693 (2007)
Hu, K., Wu, Z., Zhong, Z., Lin, W., Sun, L., Huo, Q.: A Question-Answering Approach to Key Value Pair Extraction from Form-like Document Images. arXiv preprint arXiv:2304.07957 (2023)
Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y., Manmatha, R.: DocFormer: end-to-end transformer for document understanding. In: ICCV (2021)
Powalski, R., Borchmann, Ł, Jurkiewicz, D., Dwojak, T., Pietruszka, M., Pałka, G.: Going Full-TILT boogie on document understanding with text-image-layout transformer. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 732–747. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86331-9_47
Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems (2017)
Park, S., et al.: CORD: a consolidated receipt dataset for post-OCR parsing. In: Workshop on Document Intelligence at NeurIPS 2019 (2019)
Jaume, G., Ekenel, H.K., Thiran, J.P.: FUNSD: a dataset for form understanding in noisy scanned documents. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW) (2019)
Huang, J., et al.: Few-shot named entity recognition: a comprehensive study. arXiv preprint arXiv:2012.14978 (2020)
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation (1989)
Zhou, D.W., Ye, H.J., Ma, L., Xie, D., Pu, S., Zhan, D.C.: Few-shot class-incremental learning by sampling multi-phase tasks. IEEE Trans. Pattern Anal. Mach. Intell. (2022)
Monaikul, N., Castellucci, G., Filice, S., Rokhlenko, O.: Continual learning for named entity recognition. In: AAAI (2021)
Chen, L., Moschitti, A.: Transfer learning for sequence labeling using source model and target data. In: AAAI (2019)
Tao, X., Hong, X., Chang, X., Dong, S., Wei, X., Gong, Y.: Few-shot class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12183–12192 (2020)
Cheraghian, A., Rahman, S., Fang, P., Roy, S.K., Petersson, L., Harandi, M.: Semantic-aware knowledge distillation for few-shot class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
Greenberg, N., Bansal, T., Verga, P., McCallum, A.: Marginal likelihood training of BiLSTM-CRF for biomedical named entity recognition from disjoint label sets. In: EMNLP 2018, pp. 2824–2829 (2018)
Tong, M., et al.: Learning from miscellaneous other-class words for few-shot named entity recognition. arXiv preprint arXiv:2106.15167 (2021)
BPI Challenge 2019 (2019). https://icpmconference.org/2019/icpm-2019/contests-challenges/bpi-challenge-2019/
Rizk, Y., et al.: A conversational digital assistant for intelligent process automation. In: Asatiani, A., et al. (eds.) BPM 2020. LNBIP, vol. 393, pp. 85–100. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58779-6_6
Rizk, Y., Venkateswaran, P., Isahagian, V., Muthusamy, V., Talamadupula, K.: Can you teach robotic process automation bots new tricks? In: Marrella, A., et al. (eds.) BPM 2022. LNBIP, vol. 459, pp. 246–259. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16168-1_16
Weske, M.: Business process management methodology. In: Weske, M. (ed.) Business Process Management: Concepts, Languages, Architectures, pp. 373–388. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28616-2_8
Huo, S., Völzer, H., Reddy, P., Agarwal, P., Isahagian, V., Muthusamy, V.: Graph autoencoders for business process anomaly detection. In: Polyvyanyy, A., Wynn, M.T., Van Looy, A., Reichert, M. (eds.) BPM 2021. LNCS, vol. 12875, pp. 417–433. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85469-0_26
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kumar, R., Goyal, S., Verma, A., Isahagian, V. (2024). ProtoNER: Few Shot Incremental Learning for Named Entity Recognition Using Prototypical Networks. In: De Weerdt, J., Pufahl, L. (eds) Business Process Management Workshops. BPM 2023. Lecture Notes in Business Information Processing, vol 492. Springer, Cham. https://doi.org/10.1007/978-3-031-50974-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-50974-2_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50973-5
Online ISBN: 978-3-031-50974-2
eBook Packages: Computer ScienceComputer Science (R0)