Skip to main content
SpringerLink
Log in
Book cover

International Workshop on Chatbot Research and Design

CONVERSATIONS 2019: Chatbot Research and Design pp 19–33Cite as

An Approach for Ex-Post-Facto Analysis of Knowledge Graph-Driven Chatbots – The DBpedia Chatbot

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11970))

Abstract

As chatbots are gaining popularity for simplifying access to information and community interaction, it is essential to examine whether these agents are serving their intended purpose and catering to the needs of their users. Therefore, we present an approach to perform an ex-post-facto analysis over the logs of knowledge base-driven dialogue systems. Using the DBpedia Chatbot as our case study, we inspect three aspects of the interactions, (i) user queries and feedback, (ii) the bot’s response to these queries, and (iii) the overall flow of the conversations. We discuss key implications based on our findings. All the source code used for the analysis can be found at https://github.com/dice-group/DBpedia-Chatlog-Analysis.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://chat.dbpedia.org.

  2. 2.

    https://github.com/dice-group/DBpedia-Chatlog-Analysis.

  3. 3.

    https://github.com/dbpedia/chatbot.

  4. 4.

    http://products.wolframalpha.com/api/.

  5. 5.

    https://sourceforge.net/p/dbpedia/mailman/dbpedia-discussion/.

  6. 6.

    https://sourceforge.net/p/dbpedia/mailman/dbpedia-developers/.

  7. 7.

    https://wiki.dbpedia.org/blog/meet-dbpedia-chatbot.

  8. 8.

    No unique identifiers or demographics were collected by the DBpedia Chatbot.

  9. 9.

    Inspired by https://building.lang.ai/sorry-i-didnt-get-that-how-to-understand-what-your-users-want-a90c7ca18a8f.

  10. 10.

    Applied as a preprocessing step for t-SNE algorithm, see https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html.

  11. 11.

    We refer the interested readers to also check https://stats.stackexchange.com/questions/263539/clustering-on-the-output-of-t-sne.

  12. 12.

    Using the python library langdetect https://pypi.org/project/langdetect/.

  13. 13.

    https://github.com/huggingface/neuralcoref.

References

  1. Abujabal, A., Roy, R.S., Yahya, M., Weikum, G.: ComQA: a community-sourced dataset for complex factoid question answering with paraphrase clusters (2019)

    Google Scholar 

  2. Akhtar, M., Neidhardt, J., Werthner, H.: The potential of chatbots: analysis of chatbot conversations. In: CBI (2019)

    Google Scholar 

  3. Ashktorab, Z., Jain, M., Liao, Q.V., Weisz, J.D.: Resilient chatbots: repair strategy preferences for conversational breakdowns. In: CHI (2019)

    Google Scholar 

  4. Athreya, R.G., Ngomo, A.N., Usbeck, R.: Enhancing community interactions with data-driven chatbots-the dbpedia chatbot (2018)

    Google Scholar 

  5. Both, A., Diefenbach, D., Singh, K., Shekarpour, S., Cherix, D., Lange, C.: Qanary – a methodology for vocabulary-driven open question answering systems. In: Sack, H., Blomqvist, E., d’Aquin, M., Ghidini, C., Ponzetto, S.P., Lange, C. (eds.) ESWC 2016. LNCS, vol. 9678, pp. 625–641. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-34129-3_38

    Chapter  Google Scholar 

  6. Chakraborty, N., Lukovnikov, D., Maheshwari, G., Trivedi, P., Lehmann, J., Fischer, A.: Introduction to neural network based approaches for question answering over knowledge graphs. CoRR abs/1907.09361 (2019)

    Google Scholar 

  7. Clark, K., Manning, C.D.: Deep reinforcement learning for mention-ranking coreference models. In: EMNLP (2016)

    Google Scholar 

  8. Clark, K., Manning, C.D.: Improving coreference resolution by learning entity-level distributed representations (2016)

    Google Scholar 

  9. Daiber, J., Jakob, M., Hokamp, C., Mendes, P.N.: Improving efficiency and accuracy in multilingual entity extraction (2013)

    Google Scholar 

  10. Dale, R.: The return of the chatbots. Nat. Lang. Eng. 22(5), 811–817 (2016)

    Article  Google Scholar 

  11. Diefenbach, D., Migliatti, P.H., Qawasmeh, O., Lully, V., Singh, K., Maret, P.: QAnswer: a question answering prototype bridging the gap between a considerable part of the LOD cloud and end-users (2019)

    Google Scholar 

  12. Huang, J., Zhou, M., Yang, D.: Extracting chatbot knowledge from online discussion forums. In: IJCAI, vol. 7, pp. 423–428 (2007)

    Google Scholar 

  13. Jain, M., Kumar, P., Kota, R., Patel, S.N.: Evaluating and informing the design of chatbots. In: DIS (2018)

    Google Scholar 

  14. Jenkins, M.-C., Churchill, R., Cox, S., Smith, D.: Analysis of user interaction with service oriented chatbot systems. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4552, pp. 76–83. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73110-8_9

    Chapter  Google Scholar 

  15. Keyner, S., Savenkov, V., Vakulenko, S.: Open data chatbot. In: Hitzler, P., et al. (eds.) ESWC 2019. LNCS, vol. 11762, pp. 111–115. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32327-1_22

    Chapter  Google Scholar 

  16. Lehmann, J., et al.: DBpedia - a large-scale, multilingual knowledge base extracted from Wikipedia. Semant. Web 6, 167–195 (2015)

    Article  Google Scholar 

  17. Li, C., Chen, K., Chang, Y.: When there is no progress with a task-oriented chatbot: a conversation analysis (2019)

    Google Scholar 

  18. Lommatzsch, A., Katins, J.: An information retrieval-based approach for building intuitive chatbots for large knowledge bases (2019)

    Google Scholar 

  19. Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)

    MATH  Google Scholar 

  20. McInnes, L., Healy, J., Astels, S.: HDBSCAN: hierarchical density based clustering. J. Open Source Softw. 2(11), 205 (2017)

    Article  Google Scholar 

  21. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training distributed word representations. In: LREC (2018)

    Google Scholar 

  22. Qi, P., Dozat, T., Zhang, Y., Manning, C.D.: Universal dependency parsing from scratch (2018). https://nlp.stanford.edu/pubs/qi2018universal.pdf

  23. Rivolli, A., Amaral, C., Guardão, L., de Sá, C.R., Soares, C.: KnowBots: discovering relevant patterns in chatbot dialogues. In: Kralj Novak, P., Šmuc, T., Džeroski, S. (eds.) DS 2019. LNCS (LNAI), vol. 11828, pp. 481–492. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33778-0_36

    Chapter  Google Scholar 

  24. Skjuve, M., Haugstveit, I.M., Følstad, A., Brandtzaeg, P.B.: Help! is my chatbot falling into the uncanny valley? An empirical study of user experience in human-chatbot interaction. Hum. Technol. 15, 30–54 (2019)

    Article  Google Scholar 

  25. Vakulenko, S., Garcia, J.D.F., Polleres, A., de Rijke, M., Cochez, M.: Message passing for complex question answering over knowledge graphs (2019)

    Google Scholar 

  26. Yang, Y., et al.: Multilingual universal sentence encoder for semantic retrieval. CoRR abs/1907.04307 (2019)

    Google Scholar 

Download references

Acknowledgments

This work has been supported by the Federal Ministry of Transport and Digital Infrastructure (BMVI) in the OPAL research project (grant no. 19F2028A) and LIMBO (no. 19F2029I) as well as by the German Federal Ministry of Education and Research (BMBF) within ‘KMU-innovativ: Forschung für die zivile Sicherheit’ in particular ‘Forschung für die zivile Sicherheit’ and the project SOLIDE (no. 13N14456).

Author information

Authors and Affiliations

  1. DICE Group, CS Department, Paderborn University, Paderborn, Germany

    Rricha Jalota, Axel-Cyrille Ngonga Ngomo & Ricardo Usbeck

  2. Fraunhofer IAIS, Dresden, Germany

    Priyansh Trivedi, Gaurav Maheshwari & Ricardo Usbeck

Authors
  1. Rricha Jalota
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Priyansh Trivedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gaurav Maheshwari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Axel-Cyrille Ngonga Ngomo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ricardo Usbeck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rricha Jalota .

Editor information

Editors and Affiliations

  1. SINTEF, Oslo, Norway

    Asbjørn Følstad

  2. University of Amsterdam, Amsterdam, The Netherlands

    Theo Araujo

  3. CERTH-ITI, Thessaloniki, Greece

    Symeon Papadopoulos

  4. University of Leicester, Leicester, UK

    Effie Lai-Chong Law

  5. University of Agder, Grimstad, Norway

    Ole-Christoffer Granmo

  6. University of Edinburgh, Edinburgh, UK

    Ewa Luger

  7. University of Oslo, Oslo, Norway

    Petter Bae Brandtzaeg

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jalota, R., Trivedi, P., Maheshwari, G., Ngonga Ngomo, AC., Usbeck, R. (2020). An Approach for Ex-Post-Facto Analysis of Knowledge Graph-Driven Chatbots – The DBpedia Chatbot. In: Følstad, A., et al. Chatbot Research and Design. CONVERSATIONS 2019. Lecture Notes in Computer Science(), vol 11970. Springer, Cham. https://doi.org/10.1007/978-3-030-39540-7_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-39540-7_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-39539-1

  • Online ISBN: 978-3-030-39540-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation