skip to main content
10.1145/3488560.3502196acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
short-paper

Quality Assurance of a German COVID-19 Question Answering Systems using Component-based Microbenchmarking

Published: 15 February 2022 Publication History

Abstract

Question Answering (QA) has become an often used method to retrieve data as part of chatbots and other natural-language user interfaces. In particular, QA systems of official institutions have high expectations regarding the answers computed by the system, as the provided information might be critical. In this demonstration, we use the official COVID-19 QA system that was developed together with the German Federal government to provide German citizens access to data regarding incident values, number of deaths, etc. To ensure high quality, a component-based approach was used that enables exchanging data between QA components using RDF and validating the functionality of the QA system using SPARQL. Here, we will demonstrate how our solution enables developers of QA systems to use a descriptive approach to validate the quality of their implementation before the system's deployment and also within a live environment.

Supplementary Material

MP4 File (WSDM22-wsdmde024.mp4)
Question Answering (QA) has become an often used method to retrieve data as part of chatbots and other natural-language user interfaces. In particular, QA systems of official institutions have high expectations regarding the answers computed by the system, as the provided information might be critical. In this demonstration, we use the official COVID-19 QA system that was developed together with the German Federal government to provide German citizens access to data regarding incident values, number of deaths, etc. To ensure high quality, a component-based approach was used that enables exchanging data between QA components using RDF and validating the functionality of the QA system using SPARQL. Here, we will demonstrate how our solution enables developers of QA systems to use a descriptive approach to validate the quality of their implementation before the system's deployment and also within a live environment.

References

[1]
A. Both, D. Diefenbach, K. Singh, S. Shekarpour, D. Cherix, and C. Lange. 2016. Qanary--a methodology for vocabulary-driven open question answering systems. In European Semantic Web Conference. Springer, 625--641.
[2]
A. Both, A. Perevalov, J. R. Bartsch, P. Heinze, R. Iudin, J. R. Herkner, T. Schrader, J. Wunsch, A. K. Falkenhain, and R. Gürth. 2021. A Question Answering System for retrieving German COVID-19 data driven and quality-controlled by Semantic Technology. In Joint Proceedings of the Semantics co-located events: Poster&Demo track and Workshop on Ontology-Driven Conceptual Modelling of Digital Twins co-located with SEMANTiCS 2021 (CEUR Workshop Proc., Vol. 2774). CEUR-WS.org.
[3]
A. Delpeuch. 2019. OpenTapioca: Lightweight Entity Linking for Wikidata. CoRR, Vol. abs/1904.09131 (2019). showeprint[arXiv]1904.09131 http://arxiv.org/abs/1904.09131
[4]
D. Diefenbach, V. Ló pez, K. D.Singh, and P. Maret. 2018. Core techniques of question answering systems over knowledge bases: a survey. Knowledge and Information systems, Vol. 55, 3 (2018), 529--569. https://doi.org/10.1007/s10115-017--1100-y
[5]
D. Diefenbach, K. Singh, A. Both, D. Cherix, C. Lange, and S. Auer. 2017. The Qanary Ecosystem: Getting New Insights by Composing Question Answering Pipelines. In Web Engineering - 17th International Conference, ICWE (LNCS, Vol. 10360). Springer.
[6]
M. Dubey, D. Banerjee, A. Abdelkawi, and J. Lehmann. 2019. LC-QuAD 2.0: A large dataset for complex question answering over Wikidata and DBpedia. In International Semantic Web Conference (ISWC 2019). Springer, 69--78.
[7]
Ó. Ferrández, C. Spurk, M. Kouylekov, I. Dornescu, S. Ferrández, M. Negri, R. Izquierdo, D. Tomás, C. Orasan, G. Neumann, B. Magnini, and J. L. Vicedo. 2011. The QALL-ME Framework: A specifiable-domain multilingual Question Answering architecture. Journal of Web Semantics, Vol. 9, 2 (2011), 137 -- 145.
[8]
K. Singh, A. S. Radhakrishna, A. Both, S. Shekarpour, I. Lytra, R. Usbeck, et al. 2018. Why reinvent the wheel: Let's build question answering systems together. In Proceedings of the 2018 World Wide Web Conference. 1247--1256.
[9]
A. Talmor and J. Berant. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In Proceedings of the 2018 Conference of the North American Chapter of the ACL: Human Language Technologies, Volume 1 (Long Papers). 641--651.
[10]
R. Usbeck, R. H. Gusmita, A.-C. N. Ngomo, and M. Saleem. 2018. 9th Challenge on Question Answering over Linked Data (QALD-9). In Joint proceedings of the SemDeep-4 and NLIWoD4 and 9th Question Answering over Linked Data challenge co-located with 17th International Semantic Web Conference (ISWC 2018) .
[11]
R. Usbeck, M. Röder, A.-C. N. Ngomo, C. Baron, A. Both, M. Brümmer, D. Ceccarelli, M. Cornolti, D. Cherix, B. Eickmann, P. Ferragina, C. Lemke, A. Moro, R. Navigli, F. Piccinno, G. Rizzo, H. Sack, R. Speck, R. Troncy, J. Waitelonis, and L. Wesemann. 2015. GERBIL: General Entity Annotator Benchmarking Framework. In Proceedings of the 24th International Conference on World Wide Web (WWW '15). 1133--1143.
[12]
A. P. B. Veyseh. 2016. Cross-Lingual Question Answering Using Common Semantic Space. In Proc. of TextGraphs-10: the Workshop on Graph-based Methods for Natural Language Processing . Association for Computational Linguistics, 15--19.

Cited By

View all
  • (2024)Multilingual question answering systems for knowledge graphs – a surveySemantic Web10.3233/SW-24363315:5(2089-2124)Online publication date: 9-Oct-2024
  • (2023)Natural Language Processing Adoption in Governments and Future Research Directions: A Systematic ReviewApplied Sciences10.3390/app13221234613:22(12346)Online publication date: 15-Nov-2023
  • (2023)Question Answering System Based on University Knowledge GraphComputer Supported Cooperative Work and Social Computing10.1007/978-981-99-2385-4_12(164-174)Online publication date: 13-May-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining
February 2022
1690 pages
ISBN:9781450391320
DOI:10.1145/3488560
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. ai system engineering
  2. component-oriented validation
  3. covid-19
  4. quality assurance
  5. question answering

Qualifiers

  • Short-paper

Conference

WSDM '22

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)0
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Multilingual question answering systems for knowledge graphs – a surveySemantic Web10.3233/SW-24363315:5(2089-2124)Online publication date: 9-Oct-2024
  • (2023)Natural Language Processing Adoption in Governments and Future Research Directions: A Systematic ReviewApplied Sciences10.3390/app13221234613:22(12346)Online publication date: 15-Nov-2023
  • (2023)Question Answering System Based on University Knowledge GraphComputer Supported Cooperative Work and Social Computing10.1007/978-981-99-2385-4_12(164-174)Online publication date: 13-May-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media