Skip to main content
Springer Nature Link
Log in

RuREBus: A Case Study of Joint Named Entity Recognition and Relation Extraction from E-Government Domain

  • Conference paper
  • First Online:
Analysis of Images, Social Networks and Texts (AIST 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12602))

  • 990 Accesses

Abstract

We show-case an application of information extraction methods, such as named entity recognition (NER) and relation extraction (RE) to a novel corpus, consisting of documents, issued by a state agency. The main challenges of this corpus are: 1) the annotation scheme differs greatly from the one used for the general domain corpora, and 2) the documents are written in a language other than English. Unlike expectations, the state-of-the-art transformer-based models show modest performance for both tasks, either when approached sequentially, or in an end-to-end fashion. Our experiments have demonstrated that fine-tuning on a large unlabeled corpora does not automatically yield significant improvement and thus we may conclude that more sophisticated strategies of leveraging unlabelled texts are demanded. In this paper, we describe the whole developed pipeline, starting from text annotation, baseline development, and designing a shared task in hopes of improving the baseline. Eventually, we realize that the current NER and RE technologies are far from being mature and do not overcome so far challenges like ours.

The extended notes for invited talk “When CoNLL-2003 is not Enough: are Academic NER and RE Corpora Well-Suited to Represent Real-World Scenarios?” delivered by Ivan Smurov.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The corpus is open and available online on the Ministry of Economic Development of the Russian Federation website.

  2. 2.

    https://github.com/dialogue-evaluation/RuREBus/.

  3. 3.

    Obtained after the shared task deadline.

References

  1. Anisimovich, K., Druzhkin, K., Minlos, F., Petrova, M., Selegey, V., Zuev, K.: Syntactic and semantic parser based on abbyy compreno linguistic technologies. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog” [Komp’iuternaia Lingvistika i Intellektual’nye Tehnologii: Trudy Mezhdunarodnoj Konferentsii “Dialog”], Bekasovo, Russiavol. 2, pp. 90–103 (2012)

    Google Scholar 

  2. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    Article  Google Scholar 

  3. Cardellino, C., Teruel, M., Alemany, L.A., Villata, S.: A low-cost, high-coverage legal named entity recognizer, classifier and linker. In: Proceedings of the 16th Edition of the International Conference on Artificial Intelligence and Law, pp. 9–18 (2017)

    Google Scholar 

  4. Carreras, X., Màrquez, L.: Introduction to the CoNLL-2004 shared task: Semantic role labeling. In: Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pp. 89–97. Association for Computational Linguistics, Boston (2004). https://www.aclweb.org/anthology/W04-2412

  5. Clark, K., Luong, M.T., Manning, C.D., Le, Q.: Semi-supervised sequence modeling with cross-view training. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1914–1925 (2018)

    Google Scholar 

  6. Da San Martino, G., Barrón-Cedeño, A., Wachsmuth, H., Petrov, R., Nakov, P.: SemEval-2020 task 11: detection of propaganda techniques in news articles. In: Proceedings of the 14th International Workshop on Semantic Evaluation. SemEval 2020, Barcelona, Spain (2020)

    Google Scholar 

  7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long and Short Papers), pp. 4171–4186 (2019)

    Google Scholar 

  8. Dozier, C., Kondadadi, R., Light, M., Vachher, A., Veeramachaneni, S., Wudali, R.: Named entity recognition and resolution in legal text. In: Francesconi, E., Montemagni, S., Peters, W., Tiscornia, D. (eds.) Semantic Processing of Legal Texts. LNCS (LNAI), vol. 6036, pp. 27–43. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12837-0_2

    Chapter  Google Scholar 

  9. Gordeev, D., Davletov, A., Rey, A., Akzhigitova, G., Geymbukh, G.: Relation extraction dataset for the russian language. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog” [Komp’iuternaia Lingvistika i Intellektual’nye Tehnologii: Trudy Mezhdunarodnoj Konferentsii “Dialog”] Moscow, Russia (2020)

    Google Scholar 

  10. Hendrickx, I., et al.: SemEval-2010 task 8: multi-way classification of semantic relations between pairs of nominals. In: Proceedings of the 5th International Workshop on Semantic Evaluation, pp. 33–38. Association for Computational Linguistics, Uppsala (2010). https://www.aclweb.org/anthology/S10-1006

  11. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., Weischedel, R.: Ontonotes: the 90% solution. In: Proceedings of the Human Language Technology Conference of the NAACL, Companion, NAACL-Short 2006, vol. Short Papers, p. 57–60. Association for Computational Linguistics, USA (2006)

    Google Scholar 

  12. Huang, C.C., Lu, Z.: Community challenges in biomedical text mining over 10 years: success, failure and the future. Briefings Bioinf. 17(1), 132–144 (2015)

    Article  Google Scholar 

  13. Ivanin, V., et al.: Rurebus-2020 shared task: Russian relation extraction for business. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog” [Komp’iuternaia Lingvistika i Intellektual’nye Tehnologii: Trudy Mezhdunarodnoj Konferentsii “Dialog”], Moscow, Russia (2020)

    Google Scholar 

  14. Joshi, M., Chen, D., Liu, Y., Weld, D.S., Zettlemoyer, L., Levy, O.: Spanbert: improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguist. 8, 64–77 (2020)

    Article  Google Scholar 

  15. Kuratov, Y., Arkhipov, M.: Adaptation of deep bidirectional multilingual transformers for russian language. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog” [Komp’iuternaia Lingvistika i Intellektual’nye Tehnologii: Trudy Mezhdunarodnoj Konferentsii “Dialog”], pp. 333–339 (2019)

    Google Scholar 

  16. Kutuzov, A., Kuzmenko, E.: WebVectors: a toolkit for building web interfaces for vector semantic models. In: Ignatov, D.I., et al. (eds.) AIST 2016. CCIS, vol. 661, pp. 155–161. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52920-2_15

    Chapter  Google Scholar 

  17. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition, pp. 260–270 (2016)

    Google Scholar 

  18. Leitner, E., Rehm, G., Moreno-Schneider, J.: Fine-grained named entity recognition in legal documents. In: Acosta, M., Cudré-Mauroux, P., Maleshkova, M., Pellegrini, T., Sack, H., Sure-Vetter, Y. (eds.) SEMANTiCS 2019. LNCS, vol. 11702, pp. 272–287. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33220-4_20

    Chapter  Google Scholar 

  19. Leitner, E., Rehm, G., Moreno-Schneider, J.: A dataset of german legal documents for named entity recognition. arXiv preprint arXiv:2003.13016 (2020)

  20. Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol. 1: Long Papers, pp. 1064–1074 (2016)

    Google Scholar 

  21. Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of NAACL-HLT, pp. 2227–2237 (2018)

    Google Scholar 

  22. Shen, Y., Yun, H., Lipton, Z.C., Kronrod, Y., Anandkumar, A.: Deep active learning for named entity recognition. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 252–256 (2017)

    Google Scholar 

  23. Starostin, A., et al.: Factrueval 2016: Evaluation of named entity recognition and fact extraction systems for Russian. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog” [Komp’iuternaia Lingvistika i Intellektual’nye Tehnologii: Trudy Mezhdunarodnoj Konferentsii “Dialog”] pp. 702–720 (2016)

    Google Scholar 

  24. Stenetorp, P., Pyysalo, S., Topić, G., Ohta, T., Ananiadou, S., Tsujii, J.: Brat: a web-based tool for NLP-assisted text annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 102–107. Association for Computational Linguistics (2012)

    Google Scholar 

  25. Strauss, B., Toma, B., Ritter, A., De Marneffe, M.C., Xu, W.: Results of the wnut16 named entity recognition shared task. In: Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pp. 138–144 (2016)

    Google Scholar 

  26. Teruel, M., Cardellino, C., Cardellino, F., Alemany, L.A., Villata, S.: Legal text processing within the mirel project. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)

    Google Scholar 

  27. Tjong Kim Sang, E.F., De Meulder, F.: Introduction to the conll-2003 shared task: Language-independent named entity recognition. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, CONLL 2003, vol. 4, p. 142–147. Association for Computational Linguistics (2003)

    Google Scholar 

  28. Walker, C., Strassel, S., Medero, J., Maeda, K.: ACE 2005 Multilingual Training Corpus. LDC2006T06. Philadelphia: Linguistic Data Consortium (2006)

    Google Scholar 

  29. Weber, L., Münchmeyer, J., Rocktäschel, T., Habibi, M., Leser, U.: Huner: improving biomedical NER with pretraining. Bioinformatics 36(1), 295–302 (2020)

    Article  Google Scholar 

  30. Wu, S., He, Y.: Enriching pre-trained language model with entity information for relation classification. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2361–2364 (2019)

    Google Scholar 

  31. Yang, Z., Salakhutdinov, R., Cohen, W.W.: Transfer learning for sequence tagging with hierarchical reccurent networks. arXiv preprint arXiv:1703.06345 (2017)

  32. Zhang, Y., Zhong, V., Chen, D., Angeli, G., Manning, C.D.: Position-aware attention and supervised data improve slot filling. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pp. 35–45 (2017). https://nlp.stanford.edu/pubs/zhang2017tacred.pdf

  33. Zuev, K.A., Indenbom, M.E., Judina, M.V.: Statistical machine translation with linguistic language model. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog” [Komp’iuternaia Lingvistika i Intellektual’nye Tehnologii: Trudy Mezhdunarodnoj Konferentsii “Dialog”], Bekasovo, Russia, vol. 2, pp. 164–172 (2013)

    Google Scholar 

Download references

Acknowledgements

Work on maintenance of the annotation system, discussions of results, and manuscript preparation was carried out by Elena Tutubalina, Vladimir Ivanov, Tatiana Batura and supported by the Russian Science Foundation grant no. 20-11-20166. Ekaterina Artemova and Veronika Sarkisyan worked on text annotation, discussions of results, and manuscript. Their work was supported by the framework of the HSE University Basic Research Program and Russian Academic Excellence Project “5–100”.

Author information

Authors and Affiliations

  1. ABBYY, Milpitas, USA

    Vitaly Ivanin & Ivan Smurov

  2. Moscow Institute of Physics and Technology, Dolgoprudny, Russia

    Vitaly Ivanin & Ivan Smurov

  3. National Research University Higher School of Economics, Moscow, Russia

    Ekaterina Artemova & Veronika Sarkisyan

  4. Novosibirsk State University, Novosibirsk, Russia

    Tatiana Batura

  5. Innopolis University, Innopolis, Russia

    Vladimir Ivanov

  6. Kazan Federal University, Kazan, Russia

    Elena Tutubalina

  7. Lomonosov Moscow State University, Moscow, Russia

    Tatiana Batura, Vladimir Ivanov & Elena Tutubalina

Authors
  1. Vitaly Ivanin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ekaterina Artemova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tatiana Batura
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vladimir Ivanov
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Veronika Sarkisyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Elena Tutubalina
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ivan Smurov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ivan Smurov .

Editor information

Editors and Affiliations

  1. RWTH Aachen University, Aachen, Germany

    Wil M. P. van der Aalst

  2. University of Ljubljana, Ljubljana, Slovenia

    Vladimir Batagelj

  3. National Research University Higher School of Economics, Moscow, Russia

    Dmitry I. Ignatov

  4. Krasovskii Institute of Mathematics and Mechanics, Yekaterinburg, Russia

    Michael Khachay

  5. National Research University Higher School of Economics, St. Petersburg, Russia

    Olessia Koltsova

  6. University of Oslo, Oslo, Norway

    Andrey Kutuzov

  7. National Research University Higher School of Economics, Moscow, Russia

    Sergei O. Kuznetsov

  8. National Research University Higher School of Economics, Moscow, Russia

    Irina A. Lomazova

  9. Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  10. LORIA, Vandœuvre lès Nancy, France

    Amedeo Napoli

  11. Skolkovo Institute of Science and Technology, Moscow, Russia

    Alexander Panchenko

  12. University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

  13. Università Ca' Foscari Venezia, Venice, Italy

    Marcello Pelillo

  14. National Research University Higher School of Economics, Nizhny Novgorod, Russia

    Andrey V. Savchenko

  15. Kazan Federal University, Kazan, Russia

    Elena Tutubalina

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ivanin, V. et al. (2021). RuREBus: A Case Study of Joint Named Entity Recognition and Relation Extraction from E-Government Domain. In: van der Aalst, W.M.P., et al. Analysis of Images, Social Networks and Texts. AIST 2020. Lecture Notes in Computer Science(), vol 12602. Springer, Cham. https://doi.org/10.1007/978-3-030-72610-2_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72610-2_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72609-6

  • Online ISBN: 978-3-030-72610-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics