skip to main content
10.1145/3425174.3425230acmotherconferencesArticle/Chapter ViewAbstractPublication PagessastConference Proceedingsconference-collections
research-article

OggyBug: A Test Automation Tool in Chatbots

Published: 22 October 2020 Publication History

Abstract

Context: Motivated by the reduction in operating costs, the use of chatbots to automate customer service has been growing. Chatbots have evolved a lot in terms of technologies used as well as in the different application areas. Problem: As it is a recent technology, there is no tool offers to support chatbot test automation with the possibility of testing context information that happens in the dialogue between the chatbot and the human; the existing tools also lack facilities to integrate different data sources that are used during the tests. Objective: Propose and evaluate a new framework for chatbot testing that considers context information and allows the integration test between different data sources. Method: from the analysis of the lack of existing works reported in the literature, a framework for self-testing of chatbots called OggyBug was proposed, which was used by two chatbots development teams that provided feedback on their use. Results: Construction of the framework called OggyBug that allows implementing, manage and report the results of the execution of automatic tests for chatbots, either through an API or through a web interface, with ease of integrating different sources of information within the automation scripts. After collecting the feedback from the teams that used the framework, we can observe the ease in defining scenarios and repeating the execution of the tests. Conclusion: Testing in context information proved to be important to verify or define the information of the conversation session. The configuration of integration tests proved to be complex, due to the need to configure web services in the chatbot's actions.

References

[1]
Christian Abbet, Meryem M'hamdi, Athanasios Giannakopoulos, Robert West, Andreea Hossmann, Michael Baeriswyl, and Claudiu Musat. 2018. Churn Intent Detection in Multilingual Chatbot Conversations and Social Media. In Proceedings of the 22nd Conference on Computational Natural Language Learning. Association for Computational Linguistics, Brussels, Belgium, 161--170. https://doi.org/10.18653/v1/K18-1016
[2]
S. A. Abdul-Kader and J. Woods. 2017. Question answer system for online feedable new born Chatbot. In 2017 Intelligent Systems Conference (IntelliSys). 863--869. https://doi.org/10.1109/IntelliSys.2017.8324231
[3]
A. Argal, S. Gupta, A. Modi, P. Pandey, S. Shim, and C. Choo. 2018. Intelligent travel chatbot for predictive recommendation in echo platform. In 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC). 176--183. https://doi.org/10.1109/CCWC.2018.8301732
[4]
C. J. Baby, F. A. Khan, and J. N. Swathi. 2017. Home automation using IoT and a chatbot using natural language processing. In 2017 Innovations in Power and Advanced Computing Technologies (i-PACT). 1--6. https://doi.org/10.1109/IPACT.2017.8245185
[5]
Botium. 2019. Botium - The Selenium for Chatbots. https://botium.atlassian.net/wiki/spaces/BOTIUM/overview
[6]
Pierre Bourque, Richard E. Fairley, and IEEE Computer Society. 2014. Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0 (3rd ed.). IEEE Computer Society Press, Washington, DC, USA.
[7]
J. Bozic, O. A. Tazl, and F. Wotawa. 2019. Chatbot Testing Using AI Planning. In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest). 37--44. https://doi.org/10.1109/AITest.2019.00-10
[8]
Josip Bozic and Franz Wotawa. 2018. Security Testing for Chatbots. In Testing Software and Systems, Inmaculada Medina-Bulo, Mercedes G Merayo, and Robert Hierons (Eds.). Springer International Publishing, Cham, 33--38.
[9]
Pin-Jung Chen, I-Hung Hsu, Yi-Yao Huang, and Hung-Yi Lee. 2017. Mitigating the impact of speech recognition errors on chatbot using sequence-to-sequence model. 497--503. https://doi.org/10.1109/ASRU.2017.8268977
[10]
Dialogflow. 2019. Dialogflow - Conceitos. https://cloud.google.com/dialogflow/docs/concepts?hl=pt-br
[11]
Gartner. 2017. 4 Uses for Chatbots in the Enterprise. https://www.gartner.com/smarterwithgartner/4-uses-for-chatbots-in-the-enterprise/
[12]
Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from Dialogue after Deployment: Feed Yourself, Chatbot!. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 3667--3684. https://doi.org/10.18653/v1/P19-1358
[13]
Hill, W. Ford, and Ingrid Farreras. 2015. Real conversations with artificial intelligence: A comparison between human-human conversations and human-chatbot conversations. Computers in Human Behavior 49 (01 2015), 245--250.
[14]
Douglas Hoffman. 1999. Test Automation Architectures: Planning for Test Automation. Copyright (01 1999).
[15]
IBM. 2019. IBM Cloud Documentos - Assistente do Watson. https://cloud.ibm.com/docs/assistant?topic=assistant-intents&locale=pt-BR
[16]
IBM. 2019. IBM Cloud Functions - Plataforma Functions-as-a-Service (FaaS)com base no Apache OpenWhisk. https://cloud.ibm.com/functions/
[17]
IBM. 2019. Watson Assistant - Build an assistant that your customers actually want to use. https://www.ibm.com/cloud/watson-assistant/
[18]
ISO. 2013. ISO/IEC/IEEE 29119--1: Software and systems engineering - Software testing-Part 1: Concepts and definitions.
[19]
Jieming Ji, Qingyun Wang, Zev Battad, Jiashun Gou, Jingfei Zhou, Rahul Divekar, Craig Carlson, and Mei Si. 2017. A Two-Layer Dialogue Framework For Authoring Social Bots. Alexa Prize Proceedings (2017).
[20]
Karolina Kuligowska. 2015. Commercial Chatbot: Performance Evaluation, Usability Metrics and Quality Standards of Embodied Conversational Agents. Professionals Center for Business Research 2 (01 2015), 1--16. https://doi.org/10.18483/PCBR.22
[21]
Divya Kumar and Krishn Mishra. 2016. The Impacts ofTest Automation on Software's Cost, Quality and Time to Market. Procedia Computer Science 79 (12 2016), 8--15. https://doi.org/10.1016/j.procs.2016.03.003
[22]
D. Madhu, C. J. N. Jain, E. Sebastain, S. Shaji, and A. Ajayakumar. 2017. A novel approach for medical assistance using trained chatbot. In 2017 International Conference on Inventive Communication and Computational Technologies (ICICCT). 243--246. https://doi.org/10.1109/ICICCT.2017.7975195
[23]
Eduardo Moresi. 2003. Metodologia da Pesquisa.
[24]
Node-RED. 2019. Node-RED - Low-code programming for event-driven applications. https://nodered.org/
[25]
B. R. Ranoliya, N. Raghuwanshi, and S. Singh. 2017. Chatbot for university related FAQs. In 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI). 1525--1530. https://doi.org/10.1109/ICACCI.2017.8126057
[26]
RASA. 2019. RASA- Forms. https://rasa.com/docs/rasa/core/forms/
[27]
RASA. 2019. The Rasa Core Dialogue Engine. https://rasa.com/docs/rasa/core/about/
[28]
Elayne Ruane, Théo Faure, Ross Smith, Dan Bean, Julie Carson-Berndsen, and Anthony Ventresque. 2018. BoTest: a Framework to Test the Quality of Conversational Agents Using Divergent Input Examples. 1--2. https://doi.org/10.1145/3180308.3180373
[29]
V. Selvi, S. Saranya, K. Chidida, and R. Abarna. 2019. Chatbot and bullyfree Chat. In 2019 IEEE International Conference on System, Computation, Automation and Networking(ICSCAN). 1--5. https://doi.org/10.1109/ICSCAN.2019.8878779
[30]
M. Su, C. Wu, K. Huang, Q. Hong, and H. Wang. 2017. A chatbot using LSTM-based multi-layer embedding for elderly care. In 2017 International Conference on Orange Technologies (ICOT). 70--74. https://doi.org/10.1109/ICOT.2017.8336091
[31]
Telegram. 2019. Telegram - a new era of messaging. https://telegram.org/
[32]
Marisa Vasconcelos, Heloisa Candello, Claudio Pinhanez, and Thiago dos Santos. 2017. Bottester: Testing Conversational Systems with Simulated Users. https://doi.org/10.1145/3160504.3160584
[33]
Dietmar Winkler, Reinhard Hametner, Thomas Östreicher, and Stefan Biffl. 2010. A framework for automated testing of automation systems. IEEE Conference on Emerging Technologies and Factory Automation (ETFA), 1--4. https://doi.org/10.1109/ETFA.2010.5641264
[34]
Mengting Yan, Paul Castro, Perry Cheng, and Vatche Ishakian. 2016. Building a Chatbot with Serverless Computing. 1--4. https://doi.org/10.1145/3007203.3007217

Cited By

View all
  • (2024)Integrating Static Quality Assurance in CI Chatbot Development WorkflowsIEEE Software10.1109/MS.2024.340155141:5(60-69)Online publication date: 1-Sep-2024
  • (2022)Automating the measurement of heterogeneous chatbot designsProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507255(1491-1498)Online publication date: 25-Apr-2022
  • (2022)Asymob: a platform for measuring and clustering chatbots2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion55297.2022.9793784(16-20)Online publication date: May-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SAST '20: Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing
October 2020
126 pages
ISBN:9781450387552
DOI:10.1145/3425174
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • SBC: Brazilian Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Chatbot
  2. Ferramenta de Automação de Testes
  3. Testes de Software

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SAST 20

Acceptance Rates

Overall Acceptance Rate 45 of 92 submissions, 49%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)16
  • Downloads (Last 6 weeks)1
Reflects downloads up to 15 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Integrating Static Quality Assurance in CI Chatbot Development WorkflowsIEEE Software10.1109/MS.2024.340155141:5(60-69)Online publication date: 1-Sep-2024
  • (2022)Automating the measurement of heterogeneous chatbot designsProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507255(1491-1498)Online publication date: 25-Apr-2022
  • (2022)Asymob: a platform for measuring and clustering chatbots2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion55297.2022.9793784(16-20)Online publication date: May-2022
  • (2022)A Review of Quality Assurance Research of Dialogue Systems2022 IEEE International Conference On Artificial Intelligence Testing (AITest)10.1109/AITest55621.2022.00021(87-94)Online publication date: Aug-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media