Skip to main content
Springer Nature Link
Log in

Few-Shot Learning for Identification of COVID-19 Symptoms Using Generative Pre-trained Transformer Language Models

  • Conference paper
  • First Online:
Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1753))

Abstract

Since the onset of the COVID-19 pandemic, social media users have shared their personal experiences related to the viral infection. Their posts contain rich information of symptoms that may provide useful hints to advancing the knowledge body of medical research and supplement the discoveries from clinical settings. Identification of symptom expressions in social media text is challenging, partially due to lack of annotated data. In this study, we investigate utilizing few-shot learning with generative pre-trained transformer language models to identify COVID-19 symptoms in Twitter posts. The results of our approach show that large language models are promising in more accurately identifying symptom expressions in Twitter posts with small amount of annotation effort, and our method can be applied to other medical and health applications where abundant of unlabeled data is available.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen, E., Lerman, K., Ferrara, E.: Tracking social media discourse about the covid-19 pandemic: development of a public coronavirus twitter data set. JMIR Public Health Surveill. 6(2), e19273 (2020)

    Article  Google Scholar 

  2. Müller, M., Salathé, M., Kummervold, P.E.: COVID-Twitter-BERT: A natural language processing model to analyse covid-19 content on twitter. arXiv preprint arXiv:2005.07503 (2020)

  3. Wijeratne, S., et al.: Feature engineering for Twitter-based applications. In Feature Engineering for Machine Learning and Data Analytics, pp. 359–393 (2018)

    Google Scholar 

  4. Guo, J.W., Radloff, C.L., Wawrzynski, S.E., Cloyes, K.G.: Mining twitter to explore the emergence of COVID-19 symptoms. Public Health Nurs. 37(6), 934–940 (2020)

    Article  Google Scholar 

  5. Krittanawong, C., Narasimhan, B., Virk, H.U.H., Narasimhan, H., Wang, Z., Tang, W.W.: Insights from Twitter about novel COVID-19 symptoms. Eur. Heart J. Digital Health 1(1), 4–5 (2020)

    Article  Google Scholar 

  6. Sarker, A., Lakamana, S., HoggBremer, W., Xie, A., AlGaradi, M.A., Yang, Y.C.: Self-reported COVID-19 symptoms on Twitter: an analysis and a research resource. J. Am. Med. Inform. Assoc. 27(8), 1310–1315 (2020)

    Article  Google Scholar 

  7. Jiang, K., Zhu, M., Bernard, G.R.: Discovery of COVID-19 symptomatic experience reported by twitter users. Stud. Health Technol. Inform. 294, 664–668 (2022)

    Google Scholar 

  8. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  9. Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)

    Google Scholar 

  10. Black, S., et al.: GPT-NeoX-20b: an open-source autoregressive language model. arXiv preprint arXiv:2204.06745 (2022)

  11. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2018)

    Google Scholar 

  12. Gao, L., et al.: The pile: an 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020)

  13. Logan IV, R.L., Balažević, I., Wallace, E., Petroni, F., Singh, S., Riedel, S.: Cutting down on prompts and parameters: simple few-shot learning with language models. arXiv preprint arXiv:2106.13353 (2021)

  14. Zhu, M., Song, Y., Jin, G., Jiang, K.: Identifying personal experience tweets of medication effects using pre-trained RoBERTa language model and its updating. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pp. 127–137 (2020)

    Google Scholar 

  15. Liu, Y., et al.: RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  16. Demner-Fushman, D., Rogers, W.J., Aronson, A.R.: MetaMap lite: an evaluation of a new Java implementation of MetaMap. J. Am. Med. Inform. Assoc. 24(4), 841–844 (2017)

    Article  Google Scholar 

  17. World Health Organization: Diagnostic testing for SARS-CoV-2 (2020). https://apps.who.int/iris/bitstream/handle/10665/334254/WHO-2019-nCoV-laboratory-2020.6-eng.pdf

Download references

Author information

Authors and Affiliations

  1. Purdue University Northwest, Hammond, IN, 46323, USA

    Keyuan Jiang

  2. Tongji University, Shanghai, China

    Minghao Zhu

  3. Vanderbilt University, Nashville, TN, 37232, USA

    Gordon R. Bernard

Authors
  1. Keyuan Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minghao Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gordon R. Bernard
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Keyuan Jiang .

Editor information

Editors and Affiliations

  1. University of Sydney, Sydney, Australia

    Irena Koprinska

  2. University of Bari Aldo Moro, Bari, Italy

    Paolo Mignone

  3. University of Pisa, Pisa, Italy

    Riccardo Guidotti

  4. Warsaw University of Technology, Warsaw, Poland

    Szymon Jaroszewicz

  5. Heidelberg University, Heidelberg, Germany

    Holger Fröning

  6. UniCredit, Rome, Italy

    Francesco Gullo

  7. University of Lisbon, Lisbon, Portugal

    Pedro M. Ferreira

  8. Roche, Basel, Switzerland

    Damian Roqueiro

  9. Barcelona Supercomputing Center, Barcelona, Spain

    Gaia Ceddia

  10. Halmstad University, Halmstad, Sweden

    Slawomir Nowaczyk

  11. University of Porto, Porto, Portugal

    João Gama

  12. University of Porto, Porto, Portugal

    Rita Ribeiro

  13. UPC BarcelonaTech, Barcelona, Spain

    Ricard Gavaldà

  14. University of Naples Federico II, Naples, Italy

    Elio Masciari

  15. University of North Carolina, Charlotte, USA

    Zbigniew Ras

  16. ICAR-CNR, Rende, Italy

    Ettore Ritacco

  17. University of Pisa, Pisa, Italy

    Francesca Naretto

  18. Aalen University of Applied Sciences, Aalen, Germany

    Andreas Theissler

  19. Warsaw University of Technology, Warszaw, Poland

    Przemyslaw Biecek

  20. KU Leuven, Leuven, Belgium

    Wouter Verbeke

  21. University of Duisburg-Essen, Essen, Germany

    Gregor Schiele

  22. Graz University of Technology, Graz, Austria

    Franz Pernkopf

  23. AMD, Dublin, Ireland

    Michaela Blott

  24. UniCredit, Rome, Italy

    Ilaria Bordino

  25. UniCredit, Milan, Italy

    Ivan Luciano Danesi

  26. National Agency for New Technologies, Rome, Italy

    Giovanni Ponti

  27. Unicredit, Rome, Italy

    Lorenzo Severini

  28. University of Bari Aldo Moro, Bari, Italy

    Annalisa Appice

  29. University of Bari Aldo Moro, Bari, Italy

    Giuseppina Andresini

  30. University of Lisbon, Lisbon, Portugal

    Ibéria Medeiros

  31. University of Lisbon, Lisbon, Portugal

    Guilherme Graça

  32. Northwestern University, Chicago, USA

    Lee Cooper

  33. Roche, Basel, Switzerland

    Naghmeh Ghazaleh

  34. University of Lausanne, Lausanne, Switzerland

    Jonas Richiardi

  35. Novartis, Basel, Switzerland

    Diego Saldana

  36. Novartis, Basel, Switzerland

    Konstantinos Sechidis

  37. Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico, Milan, Italy

    Arif Canakoglu

  38. Politecnico di Milano, Milan, Italy

    Sara Pido

  39. Politecnico di Milano, Milan, Italy

    Pietro Pinoli

  40. University of Waikato, Hamilton, New Zealand

    Albert Bifet

  41. Halmstad University, Halmstad, Sweden

    Sepideh Pashami

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jiang, K., Zhu, M., Bernard, G.R. (2023). Few-Shot Learning for Identification of COVID-19 Symptoms Using Generative Pre-trained Transformer Language Models. In: Koprinska, I., et al. Machine Learning and Principles and Practice of Knowledge Discovery in Databases. ECML PKDD 2022. Communications in Computer and Information Science, vol 1753. Springer, Cham. https://doi.org/10.1007/978-3-031-23633-4_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23633-4_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23632-7

  • Online ISBN: 978-3-031-23633-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation