Skip to main content
SpringerLink
Log in

Overview of ChEMU 2022 Evaluation Campaign: Information Extraction in Chemical Patents

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13390))

Abstract

In this paper, we provide an overview of the Cheminformatics Elsevier Melbourne University (ChEMU) evaluation lab 2022, part of the Conference and Labs of the Evaluation Forum 2022 (CLEF 2022). The ChEMU campaign focuses on information extraction tasks over chemical reactions in patents. The ChEMU 2020 lab provided two information extraction tasks, named entity recognition and event extraction. The ChEMU 2021 lab introduced one more task, anaphora resolution. This year, we re-run all the three tasks with new test data. Together, the tasks support comprehensive automatic chemical patent analysis. Herein, we describe the resources created for these tasks and the evaluation methodology adopted. We also provide a brief summary of the methods employed by participants of this lab and the results obtained across 22 runs from 3 teams, finding that several submissions achieve better results than the baseline methods prepared by the organizers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://n2c2.dbmi.hms.harvard.edu/.

  2. 2.

    https://trec.nist.gov/.

  3. 3.

    https://biocreative.bioinformatics.udel.edu/.

  4. 4.

    https://2019.bionlp-ost.org/.

  5. 5.

    https://sites.google.com/site/clefehealth/.

  6. 6.

    https://chemu.eng.unimelb.edu.au/.

  7. 7.

    https://pubmed.ncbi.nlm.nih.gov/.

  8. 8.

    https://go.drugbank.com/.

  9. 9.

    https://github.com/aspuru-guzik-group/chemical_vae/tree/master/models/zinc.

  10. 10.

    http://chemu2022.eng.unimelb.edu.au/.

  11. 11.

    Reaxys® Copyright ©2022 Elsevier Life Sciences IP Limited except certain content provided by third parties. Reaxys is a trademark of Elsevier Life Sciences IP Limited, used under license. https://www.reaxys.com.

  12. 12.

    https://brat.nlplab.org/.

  13. 13.

    https://bitbucket.org/nicta_biomed/brateval/src/master/.

References

  1. Akhondi, S.A., Hettne, K.M., Van Der Horst, E., Van Mulligen, E.M., Kors, J.A.: Recognition of chemical entities: combining dictionary-based and grammar-based approaches. J. Cheminformatics 7(1), 1–11 (2015)

    Google Scholar 

  2. Akhondi, S.A., et al.: Annotated chemical patent corpus: a gold standard for text mining. PLoS One 9(9), e107477 (2014)

    Article  Google Scholar 

  3. Akhondi, S.A., et al.: Chemical entity recognition in patents by combining dictionary-based and statistical approaches. Database, 2016 (2016)

    Google Scholar 

  4. Akhondi, S.A., et al.: Automatic identification of relevant chemical compounds from patents. Database, 2019 (2019)

    Google Scholar 

  5. Arighi, C.N., et al.: Overview of the biocreative III workshop. BMC Bioinform. 12(8), 1–9 (2011). https://doi.org/10.1186/1471-2105-12-S8-S1

    Article  Google Scholar 

  6. Bregonje, M.: Patents: a unique source for scientific technical information in chemistry related industry? World Pat. Inf. 27(4), 309–315 (2005)

    Article  Google Scholar 

  7. Cohen, K.B., et al.: Coreference annotation and resolution in the colorado richly annotated full text (CRAFT) corpus of biomedical journal articles. BMC Bioinform. 18(1), 372:1–372:14 (2017). https://doi.org/10.1186/s12859-017-1775-9

  8. Fang, B., Druckenbrodt, C., Akhondi, S.A., He, J., Baldwin, T., Verspoor, K.: ChEMU-Ref: a corpus for modeling anaphora resolution in the chemical domain. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics (2021)

    Google Scholar 

  9. Farkas, R., Vincze, V., Móra, G., Csirik, J., Szarvas, G.: The conll-2010 shared task: learning to detect hedges and their scope in natural language text. In: Proceedings of the Fourteenth Conference on Computational Natural Language Learning-Shared Task, pp. 1–12 (2010)

    Google Scholar 

  10. Ghaddar, A., Langlais, P.: Wikicoref: an english coreference-annotated corpus of wikipedia articles. In: Calzolari, N. (eds.) Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, 23–28 May 2016. European Language Resources Association (ELRA) (2016). http://www.lrec-conf.org/proceedings/lrec2016/summaries/192.html

  11. He, J., et al.: ChEMU 2021: reaction reference resolution and anaphora resolution in chemical patents. In: Hiemstra, D., Moens, M.-F., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds.) ECIR 2021. LNCS, vol. 12657, pp. 608–615. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72240-1_71

    Chapter  Google Scholar 

  12. He, J., et al.: ChEMU 2020: natural language processing methods are effective for information extraction from chemical patents. Front. Res. Metrics Anal. 6, 654438 (2021)

    Article  Google Scholar 

  13. Hemati, W., Mehler, A.: Lstmvoter: chemical named entity recognition using a conglomerate of sequence labeling tools. J. Cheminformatics 11(1), 1–7 (2019)

    Article  Google Scholar 

  14. Herrero-Zazo, M., Segura-Bedmar, I., Martínez, P., Declerck, T.: The ddi corpus: an annotated corpus with pharmacological substances and drug-drug interactions. J. Biomed. Inform. 46(5), 914–920 (2013)

    Article  Google Scholar 

  15. Hettne, K.M., et al.: A dictionary to identify small molecules and drugs in free text. Bioinformatics 25(22), 2983–2991 (2009)

    Article  Google Scholar 

  16. Hu, M., Cinciruk, D., Walsh, J.M.: Improving automated patent claim parsing: dataset, system, and experiments. arXiv preprint arXiv:1605.01744 (2016)

  17. Jaidka, K., Chandrasekaran, M.K., Rustagi, S., Kan, M.Y.: Overview of the cl-scisumm 2016 shared task. In: Proceedings of the Joint Workshop on Bibliometric-Enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL), pp. 93–102 (2016)

    Google Scholar 

  18. Kelly, L., et al.: Overview of the ShARe/CLEF ehealth evaluation lab 2014. In: Kanoulas, E. (ed.) CLEF 2014. LNCS, vol. 8685, pp. 172–191. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11382-1_17

    Chapter  Google Scholar 

  19. Kim, J.D., Wang, Y., Yasunori, Y.: The genia event extraction shared task, 2013 edition-overview. In: Proceedings of the BioNLP Shared Task 2013 Workshop, pp. 8–15 (2013)

    Google Scholar 

  20. Krallinger, M., Leitner, F., Rabal, O., Vazquez, M., Oyarzabal, J., Valencia, A.: CHEMDNER: the drugs and chemical names extraction challenge. J. Cheminformatics 7(1), 1–11 (2015)

    Article  Google Scholar 

  21. Krallinger, M., et al.: Overview of the biocreative vi chemical-protein interaction track. In: Proceedings of the Sixth Biocreative Challenge Evaluation Workshop, vol. 1, pp. 141–146 (2017)

    Google Scholar 

  22. Krallinger, M., Rabal, O., Lourenco, A., Oyarzabal, J., Valencia, A.: Information retrieval and text mining technologies for chemistry. Chem. Rev. 117(12), 7673–7761 (2017)

    Article  Google Scholar 

  23. Krallinger, M., et al.: Overview of the chemdner patents task. In: Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, pp. 63–75 (2015)

    Google Scholar 

  24. Lee, K., He, L., Lewis, M., Zettlemoyer, L.: End-to-end neural coreference resolution. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, 9–11 September 2017. pp. 188–197. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/d17-1018

  25. Lee, K., He, L., Zettlemoyer, L.: Higher-order coreference resolution with coarse-to-fine inference. In: Walker, M.A., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, 1–6 June 2018, Volume 2 (Short Papers), pp. 687–692. Association for Computational Linguistics (2018). https://doi.org/10.18653/v1/n18-2108

  26. Li, Y., et al.: Overview of ChEMU 2021: reaction reference resolution and anaphora resolution in chemical patents. In: Candan, K.S. (ed.) CLEF 2021. LNCS, vol. 12880, pp. 292–307. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85251-1_20

    Chapter  Google Scholar 

  27. Liu, H., Christiansen, T., Baumgartner, W.A., Verspoor, K.: Biolemmatizer: a lemmatization tool for morphological processing of biomedical text. J. Biomed. Semant. 3(1), 1–29 (2012)

    Article  Google Scholar 

  28. Lu, Z., et al.: The gene normalization task in biocreative III. BMC Bioinf. 12(8), 1–19 (2011). https://doi.org/10.1186/1471-2105-12-S8-S2

    Article  Google Scholar 

  29. Lupu, M., et al.: Overview of the trec 2011 chemical ir track. In: TREC (2011)

    Google Scholar 

  30. MorganMorgan, A., et al.: Overview of biocreative II gene normalization. Genome Biol. 9, S3 (2008). https://doi.org/10.1186/gb-2008-9-s2-s3

    Article  Google Scholar 

  31. Muresan, S., et al.: Making every SAR point count: the development of chemistry connect for the large-scale integration of structure and bioactivity data. Drug Discov. Today 16(23–24), 1019–1030 (2011)

    Article  Google Scholar 

  32. Narayanaswamy, M., Ravikumar, K., Vijay-Shanker, K.: A biological named entity recognizer. In: Biocomputing 2003, pp. 427–438. World Scientific (2002)

    Google Scholar 

  33. Nguyen, D.Q., et al.: ChEMU: named entity recognition and event extraction of chemical reactions from patents. In: Jose, J.M. (ed.) ECIR 2020. LNCS, vol. 12036, pp. 572–579. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_74

    Chapter  Google Scholar 

  34. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., Zhang, Y.: Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In: Pradhan, S., Moschitti, A., Xue, N. (eds.) Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning - Proceedings of the Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes, EMNLP-CoNLL 2012, 13 July 2012, Jeju Island, Korea, pp. 1–40. ACL (2012). https://www.aclweb.org/anthology/W12-4501/

  35. Rebholz-Schuhmann, D., Kirsch, H., Arregui, M., Gaudan, S., Riethoven, M., Stoehr, P.: Ebimed-text crunching to gather facts for proteins from medline. Bioinformatics 23(2), e237–e244 (2007)

    Article  Google Scholar 

  36. Senger, S., Bartek, L., Papadatos, G., Gaulton, A.: Managing expectations: assessment of chemistry databases generated by automated extraction of chemical structures from patents. J. Cheminformatics 7(1), 1–12 (2015). https://doi.org/10.1186/s13321-015-0097-z

    Article  Google Scholar 

  37. Tsatsaronis, G., et al.: Bioasq: a challenge on large-scale biomedical semantic indexing and question answering. In: AAAI Fall Symposium: Information Retrieval and Knowledge Discovery in Biomedical Text. Citeseer (2012)

    Google Scholar 

  38. Tseng, Y.H., Lin, C.J., Lin, Y.I.: Text mining techniques for patent analysis. Inf. Process. Manag. 43(5), 1216–1247 (2007)

    Article  Google Scholar 

  39. Vazquez, M., Krallinger, M., Leitner, F., Valencia, A.: Text mining for drugs and chemical compounds: methods, tools and applications. Mol. Inf. 30(6–7), 506–519 (2011)

    Article  Google Scholar 

  40. Verspoor, K., et al.: Chemu dataset for information extraction from chemical patents. Mendeley Data 2, 10–17632 (2020)

    Google Scholar 

  41. Yoshikawa, H., et al.: Detecting chemical reactions in patents. In: Proceeding 17th Annual Workshop of the Australasian Language Technology Association, ALTA 2019, Sydney, Australia, 4–6 December 2019, pp. 100–110 (2019)

    Google Scholar 

  42. Zhai, Z., et al.: Improving chemical named entity recognition in patents with contextualized word embeddings. arXiv preprint arXiv:1907.02679 (2019)

Download references

Acknowledgements

We are grateful for the detailed excerption and annotation work of the domain experts that support Reaxys, and the support of Ivan Krstic, Director of Chemistry Solutions at Elsevier. Funding for the ChEMU project is provided by an Australian Research Council Linkage Project, project number LP160101469, and Elsevier. We acknowledge the support of annotators for the anaphora resolution task, Dr. Sacha Novakovic and Colleen Hui Shiuan Yeow at the University of Melbourne.

Author information

Authors and Affiliations

  1. The University of Melbourne, Melbourne, Australia

    Yuan Li, Biaoyan Fang, Jiayuan He, Hiyori Yoshikawa, Zenan Zhai, Timothy Baldwin & Karin Verspoor

  2. Elsevier BV, Amsterdam, The Netherlands

    Saber A. Akhondi & Zubair Afzal

  3. Elsevier Information Systems GmbH, Frankfurt, Germany

    Christian Druckenbrodt & Camilo Thorne

  4. RMIT University, Melbourne, Australia

    Jiayuan He & Karin Verspoor

  5. Fujitsu Limited, Tokyo, Japan

    Hiyori Yoshikawa

Authors
  1. Yuan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Biaoyan Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiayuan He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hiyori Yoshikawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Saber A. Akhondi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christian Druckenbrodt
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Camilo Thorne
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zubair Afzal
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Zenan Zhai
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Timothy Baldwin
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Karin Verspoor
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Karin Verspoor .

Editor information

Editors and Affiliations

  1. University of Bologna, Forlì, Italy

    Alberto Barrón-Cedeño

  2. University of Padua, Padova, Italy

    Giovanni Da San Martino

  3. University of Bologna, Bologna, Italy

    Mirko Degli Esposti

  4. Instituto di Scienza e Tecnologie dell' Informazione “Alessandro Faedo”, Pisa, Italy

    Fabrizio Sebastiani

  5. University of Glasgow, Glasgow, UK

    Craig Macdonald

  6. University Milano-Bicocca, Milan, Italy

    Gabriella Pasi

  7. TU Wien, Vienna, Austria

    Allan Hanbury

  8. Leipzig University, Leipzig, Germany

    Martin Potthast

  9. University of Padua, Padova, Italy

    Guglielmo Faggioli

  10. University of Padua, Padova, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, Y. et al. (2022). Overview of ChEMU 2022 Evaluation Campaign: Information Extraction in Chemical Patents. In: Barrón-Cedeño, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2022. Lecture Notes in Computer Science, vol 13390. Springer, Cham. https://doi.org/10.1007/978-3-031-13643-6_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-13643-6_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-13642-9

  • Online ISBN: 978-3-031-13643-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation