Skip to main content
Springer Nature Link
Log in

Question Answering Systems Based on Pre-trained Language Models: Recent Progress

  • Conference paper
  • First Online:
Intelligent Information Processing XII (IIP 2024)

Part of the book series: IFIP Advances in Information and Communication Technology ((IFIPAICT,volume 703))

Included in the following conference series:

  • 465 Accesses

Abstract

Although Pre-trained Language Model (PLM) ChatGPT as a Question-Answering System (QAS) is so successful, it is still necessary to study further the QASs based on PLMs. In this paper, we survey state-of-the-art systems of this kind, identify the issues that current researchers are concerned about, explore various PLM-based methods for addressing them, and compare their pros and cons. We also discuss the datasets used for fine-tuning the corresponding PLMs and evaluating these PLM-based methods. Moreover, we summarise the criteria for evaluating these methods and compare their performance against these criteria. Finally, based on our analysis of the state-of-the-art PLM-based methods for QA, we identify some challenges for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abacha, A.B., Agichtein, E., Pinter, Y., Demner-Fushman, D.: Overview of the medical question answering task at TREC 2017 LiveQA. In: TREC, pp. 1–12 (2017)

    Google Scholar 

  2. Abacha, A.B., Shivade, C., Demner-Fushman, D.: Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering. In: Proceedings of the 18th BioNLP Workshop and Shared Task, pp. 370–379 (2019)

    Google Scholar 

  3. Alzubi, J.A., Jain, R., Singh, A., Parwekar, P., Gupta, M.: COBERT: COVID-19 question answering system using BERT. Arabian Journal for Science and Engineering, pp. 1–11 (2021)

    Google Scholar 

  4. Artetxe, M., Ruder, S., Yogatama, D.: On the cross-lingual transferability of monolingual representations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623–4637 (2020)

    Google Scholar 

  5. Berant, J., Chou, A., Frostig, R., Liang, P.: Semantic parsing on freebase from question-answer pairs. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533–1544 (2013)

    Google Scholar 

  6. Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1247–1250 (2008)

    Google Scholar 

  7. Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)

    Google Scholar 

  8. Chada, R., Natarajan, P.: FewshotQA: a simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6081–6090 (2021)

    Google Scholar 

  9. Chen, W., Verga, P., de Jong, M., Wieting, J., Cohen, W.: Augmenting pre-trained language models with QA-memory for open-domain question answering. arXiv preprint arXiv:2204.04581 (2022)

  10. Choi, E., et al.: QuAc: question answering in context. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174–2184 (2018)

    Google Scholar 

  11. Choi, S., .: DramaQA: character-centered video story understanding with hierarchical QA. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1166–1174 (2021)

    Google Scholar 

  12. Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N.A., Gardner, M.: A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011 (2021)

  13. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019)

    Google Scholar 

  14. Do, P., Phan, T.H.: Developing a BERT based triple classification model using knowledge graph embedding for question answering system. Appl. Intell. 52(1), 636–651 (2022)

    Article  Google Scholar 

  15. Duan, K., Du, S., Zhang, Y., Lin, Y., Wu, H., Zhang, Q.: Enhancement of question answering system accuracy via transfer learning and BERT. Appl. Sci. 12(22), 11522 (2022)

    Article  Google Scholar 

  16. Duan, N.: Overview of the NLPCC-ICCPOL 2016 shared task: open domain Chinese question answering. In: Lin, C.-Y., Xue, N., Zhao, D., Huang, X., Feng, Y. (eds.) ICCPOL/NLPCC -2016. LNCS (LNAI), vol. 10102, pp. 942–948. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-50496-4_89

    Chapter  Google Scholar 

  17. Duan, X., et al.: CJRC: a reliable human-annotated benchmark dataset for Chinese judicial reading comprehension. In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds.) CCL 2019. LNCS (LNAI), vol. 11856, pp. 439–451. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32381-3_36

    Chapter  Google Scholar 

  18. Dubey, M., Banerjee, D., Abdelkawi, A., Lehmann, J.: LC-QuAD 2.0: a large dataset for complex question answering over Wikidata and DBpedia. In: Ghidini, C., et al. (eds.) ISWC 2019. LNCS, vol. 11779, pp. 69–78. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30796-7_5

    Chapter  Google Scholar 

  19. Evseev, D., Arkhipov, M.Y.: SPARQL query generation for complex question answering with BERT and BiLSTM-based model. Comput. Linguist. Intellect. Technol. 19(26), 270–282 (2020)

    Article  Google Scholar 

  20. Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., Auli, M.: ELI5: long form question answering (2019). arXiv preprint arXiv:1907.09190

  21. Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., Dasigi, P.: IIRC: a dataset of incomplete information reading comprehension questions. arXiv preprint arXiv:2011.07127 (2020)

  22. Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D.: MRQA 2019 shared task: evaluating generalization in reading comprehension. In: Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1–13 (2019)

    Google Scholar 

  23. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., Berant, J.: Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguist. 9, 346–361 (2021)

    Article  Google Scholar 

  24. Gupta, D., Kumari, S., Ekbal, A., Bhattacharyya, P.: MMQA: a multi-domain multi-lingual question-answering framework for English andHindi. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation, pp. 2777–2784 (2018)

    Google Scholar 

  25. Gupta, S., Khade, N.: BERT based multilingual machine comprehension in English and Hindi (2020). arXiv preprint arXiv:2006.01432

  26. Guven, Z.A., Unalir, M.O.: Natural language based analysis of SQuAD: an analytical approach for BERT. Expert Syst. Appl. 195, 116592 (2022)

    Article  Google Scholar 

  27. Joshi, M., Choi, E., Weld, D.S., Zettlemoyer, L.: TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611 (2017)

    Google Scholar 

  28. Khazaeli, S., et al.: A free format legal question answering system. In: Proceedings of the Natural Legal Language Processing Workshop 2021, pp. 107–113 (2021)

    Google Scholar 

  29. Khorashadizadeh, H., Monsefi, R., Foolad, S.: Attention-based convolutional neural network for answer selection using BERT. In: 2020 8th Iranian Joint Congress on Fuzzy and intelligent Systems, pp. 121–126. IEEE (2020)

    Google Scholar 

  30. Kwiatkowski, T., et al.: Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguist. 7, 453–466 (2019)

    Article  Google Scholar 

  31. Lee, D., Choi, S., Jang, Y., Zhang, B.T.: Mounting video metadata on transformer-based language model for open-ended video question answering. arXiv preprint arXiv:2108.05158 (2021)

  32. Liu, S., Huang, X.: A Chinese question answering system based on GPT. In: 2019 IEEE 10th International Conference on Software Engineering and Service Science, pp. 533–537. IEEE (2019)

    Google Scholar 

  33. Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: OK-VQA: a visual question answering benchmark requiring external knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3195–3204 (2019)

    Google Scholar 

  34. Miller, G.A.: WordNet: a lexical database forEnglish. Commun. ACM 38(11), 39–41 (1995)

    Article  Google Scholar 

  35. Nakano, R., et al.: WebGPT: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 (2021)

  36. Pereira, J., Fidalgo, R., Lotufo, R., Nogueira, R.: Visconde: Multi-document QA with GPT-3 and neural reranking. arXiv preprint arXiv:2212.09656 (2022)

  37. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018). https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf

  38. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)

    Google Scholar 

  39. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)

    MathSciNet  Google Scholar 

  40. Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for SQuAD. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789 (2018)

    Google Scholar 

  41. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392 (2016)

    Google Scholar 

  42. Tan, Y., Min, D., Li, Y., Li, W., Hu, N., Chen, Y., Qi, G.: Evaluation of ChatGPT as a question answering system for answering complex questions (2023). arXiv preprint arXiv:2303.07992

  43. Trang, N.T.M., Shcherbakov, M.: Vietnamese question answering system from multilingual BERT models to monolingual BERT model. In: 2020 9th International Conference System Modeling and Advancement in Research Trends (SMART), pp. 201–206. IEEE (2020)

    Google Scholar 

  44. Trischler, A., et al.: NewsQA: a machine comprehension dataset. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 191–200 (2017)

    Google Scholar 

  45. Trivedi, H., Balasubramanian, N., Khot, T., Sabharwal, A.: MuSiQue: multihop questions via single-hop question composition. Trans. Assoc. Comput. Linguist. 10, 539–554 (2022)

    Article  Google Scholar 

  46. Wang, H., Wu, H., Zhu, H., Miao, Y., Wang, Q., Qiao, S., Zhao, H., Chen, C., Zhang, J.: A residual LSTM and Seq2Seq neural network based on GPT for Chinese rice-related question and answer system. Agriculture 12(6), 813 (2022)

    Article  Google Scholar 

  47. Widad, A., El Habib, B.L., et al.: BERT for question answering applied on COVID-19. Procedia Comput. Sci. 198, 379–384 (2022)

    Article  Google Scholar 

  48. Wu, J., Liu, J., Luo, X.: Few-shot legal knowledge question answering system for COVID-19 epidemic. In: 2020 3rd International Conference on Algorithms, Computing and Artificial Intelligence, pp. 1–6 (2020)

    Google Scholar 

  49. Yang, W., et al.: End-to-end open-domain question answering with BERTserini. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 72–77 (2019)

    Google Scholar 

  50. Yang, Y., Yih, W.t., Meek, C.: WIKIQA: a challenge dataset for open-domain question answering. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2013–2018 (2015)

    Google Scholar 

  51. Yang, Z., et al.: An empirical study of GPT-3 for few-shot knowledge-based VQA. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 3081–3089 (2022)

    Google Scholar 

  52. Yang, Z., et al.: HotpotQA: a dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380 (2018)

    Google Scholar 

  53. Yin, J.: Research on question answering system based on BERT model. In: 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications, pp. 68–71. IEEE (2022)

    Google Scholar 

  54. Yoon, W., Lee, J., Kim, D., Jeong, M., Kang, J.: Pre-trained language model for biomedical question answering. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1168, pp. 727–740. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43887-6_64

    Chapter  Google Scholar 

  55. Zaib, M., Tran, D.H., Sagar, S., Mahmood, A., Zhang, W.E., Sheng, Q.Z.: BERT-CoQAC: BERT-based conversational question answering in context. In: Ning, L., Chau, V., Lau, F. (eds.) PAAP 2020. CCIS, vol. 1362, pp. 47–57. Springer, Singapore (2021). https://doi.org/10.1007/978-981-16-0010-4_5

    Chapter  Google Scholar 

  56. Zhang, N.N., Xing, Y.: Questions and answers on legal texts based on BERT-BiGRU. J. Phys: Conf. Ser. 1828(1), 012035 (2021)

    MathSciNet  Google Scholar 

  57. Zhao, X., Li, Z., Wu, S., Zhan, Y., Zhang, C.: Deep text matching in medical question answering system. In: ACM ICEA’21: Proceedings of the 2021 ACM International Conference on Intelligent Computing and Its Emerging Applications, pp. 134–138 (2021)

    Google Scholar 

  58. Zhou, S., Zhang, Y.: DATLMedQA: a data augmentation and transfer learning based solution for medical question answering. Appl. Sci. 11(23), 11251 (2021)

    Article  Google Scholar 

  59. Zhu, J., Wu, J., Luo, X., Liu, J.: Semantic matching based legal information retrieval system for COVID-19 pandemic. Artificial Intelligence and Law, pp. 1–30 (2023)

    Google Scholar 

  60. Zihayat, M., Etwaroo, R.: A non-factoid question answering system for prior art search. Expert Syst. Appl. 177, 114910 (2021)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Guangxi Normal University, Guilin, 541004, China

    Xudong Luo, Ying Luo & Binxia Yang

  2. Key Lab of Education Blockchain and Intelligent Technology of Ministry of Education, Guilin, 541004, China

    Xudong Luo

  3. Guangxi Key Lab of Multi-Source Information Mining and Security, Guilin, 541004, China

    Xudong Luo

Authors
  1. Xudong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Binxia Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xudong Luo .

Editor information

Editors and Affiliations

  1. Chinese Academy of Sciences, Beijing, China

    Zhongzhi Shi

  2. University of Oslo, Oslo, Norway

    Jim Torresen

  3. De Montfort University, Leicester, UK

    Shengxiang Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2024 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Luo, X., Luo, Y., Yang, B. (2024). Question Answering Systems Based on Pre-trained Language Models: Recent Progress. In: Shi, Z., Torresen, J., Yang, S. (eds) Intelligent Information Processing XII. IIP 2024. IFIP Advances in Information and Communication Technology, vol 703. Springer, Cham. https://doi.org/10.1007/978-3-031-57808-3_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-57808-3_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-57807-6

  • Online ISBN: 978-3-031-57808-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships