Skip to main content
SpringerLink
Log in
SpringerLink
Log in

A Comparative Evaluation of QA Systems over List Questions

  • Conference paper
  • First Online:
Computational Processing of the Portuguese Language (PROPOR 2016)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9727))

Abstract

The evaluation of a Question Answering system is a challenging task. In this paper we evaluate our system, LX-ListQuestion, a Web-based QA System that focuses on answering list questions. We compare our system against other QA Systems and the results were analyzed in two ways: (i) the quantitative evaluation of answers provides recall, precision and F-measure and (ii) the question coverage that indicate the usefulness of the system to the user by counting the number of questions for which the system provides at least one correct answer. The evaluation brings interesting results that points to a certain degree of complementary between different approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://trec.nist.gov/.

  2. 2.

    http://www.linguateca.pt/Pagico/.

References

  1. Branco, A., Rodrigues, L., Silva, J., Silveira, S.: Real-time open-domain QA on the Portuguese web. In: Geffner, H., Prada, R., Machado Alexandre, I., David, N. (eds.) IBERAMIA 2008. LNCS (LNAI), vol. 5290, pp. 322–331. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  2. Branco, A., Rodrigues, L., Silva, J., Silveira, S.: XisQuê: an online QA service for Portuguese. In: Teixeira, A., Lima, V.L.S., Oliveira, L.C., Quaresma, P. (eds.) PROPOR 2008. LNCS (LNAI), vol. 5190, pp. 232–235. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  3. Ferreira, E., Balsa, J., Branco, A.: Combining rule-based and statistical models for named entity recognition of Portuguese. In: Proceedings of Workshop em Tecnologia da Informaçãoe de Linguagem Natural, pp. 1615–1624 (2007)

    Google Scholar 

  4. Freitas, C.: A lusofonia na Wikipédia em 150 topicos. Linguamatica 4(1), 9–18 (2012)

    Google Scholar 

  5. Gonçalves, P.: Open-Domain Web-Based Multiple Document Question Answering forList Questions with Support for Temporal Restrictors. Ph.D. thesis, University of Lisbon, Lisbon, Portugal, 6 2015

    Google Scholar 

  6. Gonçalves, P., Branco, A.: Answering list questions using web as a corpus. In: Proceedings of the Demonstrations at the 14th Conference ofthe European Chapter of the Association for Computational Linguistics, pp. 81–84. Association for Computational Linguistics, Gothenburg, April 2014

    Google Scholar 

  7. Gonçalves, P., Branco, A.: Open-domain web-based list question answering with LX-listquestion. In: Proceedings of the 4th International Conference on WebIntelligence, Mining and Semantics, WIMS 2014, pp. 43:1–43:6. ACM, New York (2014)

    Google Scholar 

  8. Radev, D.R., Qi, H., Wu, H., Fan, W.: Evaluating web-based question answering systems. In: Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002. European Language Resources Association, Las Palmas, 29-31 May 2002

    Google Scholar 

  9. Rodrigues, R., Oliveira, H.: Uma abordagem ao páigico baseada no processamento e anáilise desintagmas dos tópicos. Linguamatica 4(1), 31–39 (2012)

    Google Scholar 

  10. Strzalkowski, T., Harabagiu, S.: Advances in Open Domain Question Answering, 1st edn. Springer Publishing Company Incorporated, Netherlands (2007)

    Google Scholar 

  11. Voorhees, E.: Evaluating question answering system performance. In: Strzalkowski, T., Harabagiu, S. (eds.) Advances in OpenDomain Question Answering. Text, Speech and Language Technology, vol. 32, pp. 409–430. Springer, Netherlands (2006)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, University of Lisbon, Edifício C6, Faculdade de Ciências Campo Grande, 1749-016, Lisbon, Portugal

    Patricia Nunes Gonçalves & António Horta Branco

Authors
  1. Patricia Nunes Gonçalves
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. António Horta Branco
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to António Horta Branco .

Editor information

Editors and Affiliations

  1. Universidade de Lisbon, Portugal

    João Silva

  2. ISCTE-IUL, Lisbon, Portugal

    Ricardo Ribeiro

  3. Universidade de Évora, Évora, Portugal

    Paulo Quaresma

  4. Universidade de Caxias do Sul, Caxias do Suö, Brazil

    André Adami

  5. Universidade de Lisbon, Lisboa, Portugal

    António Branco

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Gonçalves, P.N., Branco, A.H. (2016). A Comparative Evaluation of QA Systems over List Questions. In: Silva, J., Ribeiro, R., Quaresma, P., Adami, A., Branco, A. (eds) Computational Processing of the Portuguese Language. PROPOR 2016. Lecture Notes in Computer Science(), vol 9727. Springer, Cham. https://doi.org/10.1007/978-3-319-41552-9_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-41552-9_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-41551-2

  • Online ISBN: 978-3-319-41552-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation