Skip to main content
SpringerLink
Log in

A systematic review of question answering systems for non-factoid questions

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Question Answering (QA) is a field of study addressed to develop automatic methods for answering questions expressed in natural language. Recently, the emergence of the new generation of intelligent assistants, such as Siri, Alexa, and Google Assistant, has intensified the importance of an effective and efficient QA system able to handle questions with different complexities. Regarding the type of question to be answered, QA systems have been divided into two sub-areas: (i) factoid questions that require a single fact – e.g., a name of a person or a date, and (ii) non-factoid questions that need a more complex answer – e.g., descriptions, opinions, or explanations. While factoid QA systems have overcome human performance on some benchmarks, automatic systems for answering non-factoid questions remain a challenge and an open research problem. This work provides an overview of recent research addressing non-factoid questions. It focuses on which methods have been applied in each task, the data sets available, challenges and limitations, and possible research directions. From a total of 455 recent studies, we selected 75 papers based on our quality control system and exclusion criteria for an in-depth analysis. This systematic review helped to answer what are the tasks and methods involved in non-factoid, what are the data sets available, what the limitations are, and what is the recommendations for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset (https://rajpurkar.github.io/SQuAD-explorer/)

References

  • Agichtein, E., Carmel, D., Pelleg, D., Pinter, Y., & Harman, D. (2015). Overview of the trec 2015 liveqa track.. In TREC.

  • Bae, K., & Ko, Y. (2019). Efficient question classification and retrieval using category information and word embedding on cQA services. Journal of Intelligent Information Systems, 53(1), 27–49. https://doi.org/10.1007/s10844-019-00556-x.

    Article  Google Scholar 

  • Bau, D., Liu, S., Wang, T., Zhu, J-Y, & Torralba, A. (2020). Rewriting a deep generative model. In A Vedaldi, H Bischof, T Brox, & J-M Frahm (Eds.) Computer Vision – ECCV 2020, pp 351–369. Springer International Publishing. Cham.

  • Ben Abacha, A, & Zweigenbaum, P. (2015). Means: A medical question-answering system combining nlp techniques and semantic web technologies. Information Processing & Management, 51 (5), 570–594. https://doi.org/10.1016/j.ipm.2015.04.006. https://www.sciencedirect.com/science/article/pii/S0306457315000515.

    Article  Google Scholar 

  • Bondarenko, A., Braslavski, P., Völske, M, Aly, R., Fröbe, M, Panchenko, A., Biemann, C., Stein, B., & Hagen, M. (2020). Comparative web search questions. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM ’20, pp 52–60. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3336191.3371848.

  • Calijorne Soares, M.A., & Parreiras, F.S. (2020). A literature review on question answering techniques, paradigms and systems. Journal of King Saud University - Computer and Information Sciences, 32(6), 635–646. https://doi.org/10.1016/j.jksuci.2018.08.005.

    Article  Google Scholar 

  • Chali, Y., Hasan, S.A., & Mojahid, M. (2015). A reinforcement learning formulation to the complex question answering problem. Information Processing & Management, 51(3), 252–272. https://doi.org/10.1016/j.ipm.2015.01.002, https://www.sciencedirect.com/science/article/pii/S0306457315000035.

    Article  Google Scholar 

  • Cohen, D., Yang, L., & Croft, W.B. (2018). WikiPassageQA: A benchmark collection for research on non-factoid answer passage retrieval. 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, pp 1165–1168. https://doi.org/10.1145/3209978.3210118.

  • Corbin, J., & Strauss, A. (2014). Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.

  • Cortes, E., Woloszyn, V., Binder, A., Himmelsbach, T., Barone, D., & Möller, S (May 2020). An empirical comparison of question classification methods for question answering systems. In Proceedings of the 12th Language Resources and Evaluation Conference, pp 5408–5416. European Language Resources Association, Marseille, France. https://www.aclweb.org/anthology/2020.lrec-1.665.

  • Denyer, D., & Tranfield, D. (2009). Producing a systematic review. Sage Publications Ltd, 671—689.

  • Dimitrakis, E., Sgontzos, K., & Tzitzikas, Y. (2019). A survey on question answering systems over linked data and documents. Journal of Intelligent Information Systems, 55, 233–259.

    Article  Google Scholar 

  • Dybå, T, & Dingsøyr, T (2008). Empirical studies of agile software development: A systematic review. Information and software technology, 50(9-10), 833–859.

    Article  Google Scholar 

  • Hazrina, S., Sharef, N.M., Ibrahim, H., Murad, M.A.A., & Noah, S.A.M. (2017). Review on the advancements of disambiguation in semantic question answering system. Information Processing & Management, 53 (1), 52–69. https://doi.org/10.1016/j.ipm.2016.06.006, https://www.sciencedirect.com/science/article/pii/S0306457316302102.

    Article  Google Scholar 

  • Hermjakob, U., Echihabi, A., & Marcu, D. (2002). Natural language based reformulation resource and web exploitation for question answering. In Proceedings of TREC, 11. Citeseer.

  • Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., & Welch, V.A. (2019). Cochrane handbook for systematic reviews of interventions. New York: John Wiley & Sons.

    Book  Google Scholar 

  • Joshi, M., Chen, D., Liu, Y., Weld, D.S., Zettlemoyer, L., & Levy, O. (2020). SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8, 64–77. https://doi.org/10.1162/tacl_a_00300.

    Article  Google Scholar 

  • Khan, K.S., Kunz, R., Kleijnen, J., & Antes, G. (2003). Five steps to conducting a systematic review. Journal of the royal society of medicine, 96(3), 118–121.

    Article  Google Scholar 

  • Khushhal, S., Majid, A., Abbas, S.A., Nadeem, M.S.A., & Shah, S. (2020). Question retrieval using combined queries in community question answering. Journal of Intelligent Information Systems, 55, 307–327. https://doi.org/10.1007/s10844-020-00612-x.

    Article  Google Scholar 

  • Kodra, L., & Kajo, E. (2017). Question Answering Systems: A Review on Present Developments, Challenges and Trends. International Journal of Advanced Computer Science and Applications, 8(9), 217–224. https://doi.org/10.14569/ijacsa.2017.080931.

    Article  Google Scholar 

  • Kolomiyets, O., & Moens, M.F. (2011). A survey on question answering technology from an information retrieval perspective. Information Sciences, 181(24), 5412–5434. https://doi.org/10.1016/j.ins.2011.07.047.

    Article  MathSciNet  Google Scholar 

  • Liu, Y., Yi, X., Chen, R., & Song, Y. (2016). A Survey on Frameworks and Methods of Question Answering. Proceedings - 2016 3rd International Conference on Information Science and Control Engineering, ICISCE 2016, pp 115–119. https://doi.org/10.1109/ICISCE.2016.35.

  • Malviya, M., & Soni, M. (2020). Question answering schemes: A review. International Journal of Scientific Research & Engineering Trends, 6(4), 2641–2648.

    Google Scholar 

  • Mishra, A., & Jain, S.K. (2016). A survey on question answering systems with classification. Journal of King Saud University - Computer and Information Sciences, 28(3), 345–361. https://doi.org/10.1016/j.jksuci.2014.10.007.

    Article  Google Scholar 

  • Noraset, T., Lowphansirikul, L., & Tuarob, S. (2021). Wabiqa: A wikipedia-based thai question-answering system. Information Processing & Management, 58 (1), 102431. https://doi.org/10.1016/j.ipm.2020.102431.

    Article  Google Scholar 

  • Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan—a web and mobile app for systematic reviews. Systematic reviews, 5(1), 210.

    Article  Google Scholar 

  • Papadakis, M., & Tzitzikas, Y. (2015). Answering keyword queries through cached subqueries in best match retrieval models. Journal of Intelligent Information System, 44(1), 67–106. https://doi.org/10.1007/s10844-014-0330-7.

    Article  Google Scholar 

  • Seers, K. (2012). Qualitative data analysis. Evidence-based nursing, 15(1), 2–2.

    Article  Google Scholar 

  • Shah, A.A., Ravana, S.D., Hamid, S., & Ismail, M.A. (2019). Accuracy evaluation of methods and techniques in Web-based question answering systems: a survey. Knowledge and Information Systems, 58(3), 611–650. https://doi.org/10.1007/s10115-018-1203-0.

    Article  Google Scholar 

  • Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M.W., & Keutzer, K. (2020). Q-bert: Hessian based ultra low precision quantization of bert. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8815–8821. https://doi.org/10.1609/aaai.v34i05.6409, https://ojs.aaai.org/index.php/AAAI/article/view/6409.

    Article  Google Scholar 

  • Specia, L., Scarton, C., & Paetzold, G.H. (2018). Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1), 1–162.

    Article  Google Scholar 

  • Sultana, T., & Badugu, S. (2020). A review on different question answering system approaches. In S.C. Satapathy, K.S. Raju, K. Shyamala, D.R. Krishna, & M.N. Favorskaya (Eds.) Advances in Decision Sciences, Image Processing, Security and Computer Vision, pp 579–586. Springer International Publishing, Cham.

  • Surdeanu, M., Ciaramita, M., & Zaragoza, H. (2008). Learning to rank answers on large online QA collections. In ACL-08: HLT - 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pp 719–727.

  • Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British journal of management, 14(3), 207–222.

    Article  Google Scholar 

  • Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017). Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 189–198. Association for Computational Linguistics, Vancouver, Canada. https://www.aclweb.org/anthology/P17-1018.

  • Wu, Y., Hori, C., Kashioka, H., & Kawai, H. (2015). Leveraging social Q&A collections for improving complex question answering. Computer Speech and Language, 29(1), 1–19. https://doi.org/10.1016/j.csl.2014.06.001.

    Article  Google Scholar 

  • Yan, Z., & Zhou, J. (2015). Optimal answerer ranking for new questions in community question answering. Information Processing & Management, 51 (1), 163–178. https://doi.org/10.1016/j.ipm.2014.07.009.

    Article  Google Scholar 

  • Yang, L., Ai, Q., Spina, D., Chen, R.C., Pang, L., Bruce Croft, W., Guo, J., & Scholer, F. (2016). Beyond factoid QA: Effective methods for non-factoid answer sentence retrieval. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9626, pp 115–128. Springer Verlag.

  • Yogish, D., Manjunath, T.N., & Hegadi, R.S. (2018). Survey on trends and methods of an intelligent answering system. International Conference on Electrical, Electronics, Communication Computer Technologies and Optimization Techniques, ICEECCOT 2017, 2018-Janua:346–353. https://doi.org/10.1109/ICEECCOT.2017.8284526.

  • Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning based natural language processing [review article]. IEEE Computational Intelligence Magazine, 13(3), 55–75. https://doi.org/10.1109/MCI.2018.2840738.

    Article  Google Scholar 

Download references

Funding

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

Author information

Authors and Affiliations

  1. Federal University of Rio Grande do Sul, Porto Alegre, Brazil

    Eduardo Gabriel Cortes & Dante Barone

  2. Technische Universität Berlin, Berlin, Germany

    Vinicius Woloszyn & Sebastian Möller

  3. CIDEHUS, Évora University, Évora, Portugal

    Renata Vieira

Authors
  1. Eduardo Gabriel Cortes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vinicius Woloszyn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dante Barone
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sebastian Möller
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Renata Vieira
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The idea of the systematic review on Non-factoid Question Answering came from the author Eduardo G. Cortes. All authors contributed to the study design. The survey of the analyzed studies and their analysis was carried out by Eduardo G. Cortes and Vinicius Woloszyn with Dante Barone’s supervision, Renata Vieira and Sebastian Möller. The first draft of the manuscript was written, evaluated, and corrected by all authors Eduardo G. Cortes, Vinicius Woloszyn, Dante Barone, Renata Vieira, and Sebastian Möller. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Eduardo Gabriel Cortes.

Ethics declarations

Consent to participate

There is the consent of all authors.

Consent for Publication

There is the consent of all authors.

Conflicts of Interest/Competing Interests

There are no conflicts or competing interests. We ensure that this manuscript has a novel contribution and has not been published or submitted to any other publisher before.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -Brasil (CAPES) - Finance Code 001.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cortes, E.G., Woloszyn, V., Barone, D. et al. A systematic review of question answering systems for non-factoid questions. J Intell Inf Syst 58, 453–480 (2022). https://doi.org/10.1007/s10844-021-00655-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-021-00655-8

Keywords

Search

Navigation