Skip to main content
Springer Nature Link
Log in

Identifying Topics of Scientific Articles with BERT-Based Approaches and Topic Modeling

  • Conference paper
  • First Online:
Trends and Applications in Knowledge Discovery and Data Mining (PAKDD 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12705))

Included in the following conference series:

  • 2380 Accesses

Abstract

This paper describes neural models developed for the First Workshop on Scope Detection of the Peer Review Articles shared task collocated with PAKDD 2021. The aim of the task is to identify topics or category of scientific abstracts. We investigate the use of several fine-tuned language representation models pretrained on different large-scale corpora. In addition, we conduct experiments on combining BERT-based models and document topic vectors for scientific text classification. The topic vectors are obtained using LDA topic modeling. The topic-informed soft voting ensemble of neural networks achieved F1-score of 93.82%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The source code for our models is available at: https://github.com/SDPRA-2021/shared-task/tree/main/utmn.

References

  1. Aluru, S.S., et al.: Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465 (2020)

  2. Beltagy, I., Lo, K., Cohan, A.: SciBERT: a pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)

  3. Bidulya, Y.: An Approach to the development of software for effective search of scientific articles. In: 2018 3rd Russian-Pacific Conference on Computer Technology and Applications (RPC), pp. 1–4 (2018). https://doi.org/10.1109/rpc.2018.8482164

  4. Chandrasekaran, M.K., et al.: Overview and insights from the shared tasks at scholarly document processing 2020: CL-SciSumm, LaySumm and LongSumm. In: Proceedings of the First Workshop on Scholarly Document Processing, pp. 214–224 (2020)

    Google Scholar 

  5. Chollet, F., et al.: Keras: the python deep learning library. Astrophysics Source Code Library, ascl: 1806.022 (2018)

    Google Scholar 

  6. Cox, J., Harper, C.A., de Waard, A.: Optimized machine learning methods predict discourse segment type in biological research articles. In: González-Beltrán, A., Osborne, F., Peroni, S., Vahdati, S. (eds.) SAVE-SD 2017-2018. LNCS, vol. 10959, pp. 95–109. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01379-0_7

    Chapter  Google Scholar 

  7. Devlin, J., et al.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  8. Fisas, B., Ronzano, F., Saggion, H.: A multi-layered annotated corpus of scientific papers. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pp. 3081–3088 (2016)

    Google Scholar 

  9. Jaidka, K., et al.: Insights from CL-SciSumm 2016: the faceted scientific document summarization shared task. Int. J. Digit. Libr. 192, 163–171 (2018)

    Article  Google Scholar 

  10. Gábor, K., et al.: Semeval-2018 task 7: semantic relation extraction and classification in scientific papers. In: Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 679–688 (2018)

    Google Scholar 

  11. Gordon, J., et al.: Modeling concept dependencies in a scientific corpus. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 866–875 (2016)

    Google Scholar 

  12. Ghosal, T., Verma, R., Ekbal, A., Saha, S., Bhattacharyya, P.: An empirical study of importance of different sections in research articles towards ascertaining their appropriateness to a Journal. In: Ishita, E., Pang, N.L.S., Zhou, L. (eds.) ICADL 2020. LNCS, vol. 12504, pp. 407–415. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64452-9_38

    Chapter  Google Scholar 

  13. Gipp, B., Meuschke, N., Breitinger, C.: Citation based plagiarism detection: practicability on a large scale scientific corpus. J. Am. Soc. Inf. Sci. 658, 1527–1540 (2014). https://doi.org/10.1002/asi.23228

    Article  Google Scholar 

  14. Glazkova, A.: A comparison of synthetic oversampling methods for multi-class text classification. arXiv preprint arXiv:2008.04636 (2020)

  15. Glazkova, A., Glazkov, M., Trifonov, T.: g2tmn at Constraint@AAAI2021: exploiting CT-BERT and ensembling learning for COVID-19 fake news detection. In: Proceedings of the First Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation (CONSTRAINT). Springer (2021, Forthcoming)

    Google Scholar 

  16. Gu, Y., et al.: Domain-specific language model pretraining for biomedical natural language processing. arXiv preprint arXiv:2007.15779 (2020)

  17. Lemaître, G., Nogueira, F., Aridas, C.K.: Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. J. Mach. Learn. Res. 181, 559–563 (2017)

    Google Scholar 

  18. Liu, Y., et al.: Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  19. Lo, K., et al.: S2ORC: the semantic scholar open research corpus. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983 (2020)

    Google Scholar 

  20. Loper, E., Bird, S.: NLTK: the natural language toolkit. In: Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pp. 63–70 (2002)

    Google Scholar 

  21. Loshchilov I., Hutter F.: Fixing weight decay regularization in Adam. arXiv preprint arXiv:1711.05101 (2017)

  22. Müller, M., Salathé, M., Kummervold, P.E.: Covid-Twitter-BERT: a natural language processing model to analyse covid-19 content on twitter. arXiv preprint arXiv:2005.07503 (2020)

  23. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8026–8037 (2019)

    Google Scholar 

  24. Peinelt N., Nguyen D., Liakata M.: tBERT: Topic models and BERT joining forces for semantic similarity detection. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7047–7055 (2020) https://doi.org/10.18653/v1/2020.acl-main.630

  25. Radev, D.R., et al.: The ACL anthology network corpus. Lang. Resour. Eval. 474, 919–944 (2013). https://doi.org/10.3115/1699750.1699759

    Article  Google Scholar 

  26. Reddy, S., Saini, N.: Overview and insights from scope detection of the peer review articles shared tasks 2021. In: Proceedings of The First Workshop & Shared Task on Scope Detection of the Peer Review Articles (SDPRA 2021) (2021, Forthcoming)

    Google Scholar 

  27. Reddy S., Saini N.: SDPRA 2021 shared task data. Mendeley Data V1. https://doi.org/10.17632/njb74czv49.1

  28. Rehurek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (2010)

    Google Scholar 

  29. Romanov, A., Lomotin, K., Kozlova, E.: Application of natural language processing algorithms to the task of automatic classification of russian scientific texts. Data Sci. J. 181 (2019). https://doi.org/10.5334/dsj-2019-037

  30. Saggion, H., et al.: A multi-level annotated corpus of scientific papers for scientific document summarization and cross-document relation discovery. In: Proceedings of The 12th Language Resources and Evaluation Conference, pp. 6672–6679 (2020)

    Google Scholar 

  31. Soares F., Moreira V., Becker K.: A large parallel corpus of full-text scientific articles. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)

    Google Scholar 

  32. Solovyev, V., Ivanov, V., Solnyshkina, M.: Assessment of reading difficulty levels in Russian academic texts: approaches and metrics. J. Intell. Fuzzy Syst. 345, 3049–3058 (2018). https://doi.org/10.3233/jifs-169489

    Article  Google Scholar 

  33. Sun, Z., et al.: MobileBERT: a compact task-agnostic BERT for resource-limited devices. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2158–2170 (2020)

    Google Scholar 

  34. Teich, E.: Exploring a corpus of scientific texts using data mining. In: Corpus-Linguistic Applications, pp. 233–247 (2010)

    Google Scholar 

  35. Veyseh, A.P.B., et al.: Acronym identification and disambiguation shared tasks for scientific document understanding. arXiv preprint arXiv:2012.11760 (2020)

  36. Vincze, V., et al.: The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinf. 911, 1–9 (2008)

    Google Scholar 

  37. Weissenbacher, D., et al.: SemEval-2019 task 12: toponym resolution in scientific papers. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 907–916 (2019)

    Google Scholar 

  38. Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020) https://doi.org/10.18653/v1/2020.emnlp-demos.6

  39. Xia, P., Wu, S., Van Durme, B.: Which* BERT? A survey organizing contextualized encoders. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7516–7533 (2020)

    Google Scholar 

  40. Yasunaga, M., et al.: ScisummNet: a large annotated corpus and content-impact models for scientific paper summarization with citation networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 331, pp. 7386–7393 (2019)

    Google Scholar 

  41. Zakharova, I., et al.: Diagnostics of professional competence of IT students based on digital footprint data. Inf. Educ. 4, 4–11 (2020). https://doi.org/10.32517/0234-0453-2020-35-4-4-11

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Tyumen, ul. Volodarskogo 6, 625003, Tyumen, Russia

    Anna Glazkova

Authors
  1. Anna Glazkova
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anna Glazkova .

Editor information

Editors and Affiliations

  1. Microsoft, Hyderabad, India

    Manish Gupta

  2. Indian Institute of Technology Bombay, Mumbai, India

    Ganesh Ramakrishnan

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Glazkova, A. (2021). Identifying Topics of Scientific Articles with BERT-Based Approaches and Topic Modeling. In: Gupta, M., Ramakrishnan, G. (eds) Trends and Applications in Knowledge Discovery and Data Mining. PAKDD 2021. Lecture Notes in Computer Science(), vol 12705. Springer, Cham. https://doi.org/10.1007/978-3-030-75015-2_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-75015-2_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-75014-5

  • Online ISBN: 978-3-030-75015-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics