Ahmad Abdellatif
ABSTRACT
Chatbots are becoming increasingly popular due to their benefits in saving costs, time, and effort. This is due to the fact that they allow users to communicate and control different services easily through natural language. Chatbot development requires special expertise (e.g., machine learning and conversation design) that differ from the development of traditional software systems. At the same time, the challenges that chatbot developers face remain mostly unknown since most of the existing studies focus on proposing chatbots to perform particular tasks rather than their development.
Therefore, in this paper, we examine the Q&A website, Stack Overflow, to provide insights on the topics that chatbot developers are interested and the challenges they face. In particular, we leverage topic modeling to understand the topics that are being discussed by chatbot developers on Stack Overflow. Then, we examine the popularity and difficulty of those topics. Our results show that most of the chatbot developers are using Stack Overflow to ask about implementation guidelines. We determine 12 topics that developers discuss (e.g., Model Training) that fall into five main categories. Most of the posts belong to chatbot development, integration, and the natural language understanding (NLU) model categories. On the other hand, we find that developers consider the posts of building and integrating chatbots topics more helpful compared to other topics. Specifically, developers face challenges in the training of the chatbot's model. We believe that our study guides future research to propose techniques and tools to help the community at its early stages to overcome the most popular and difficult topics that practitioners face when developing chatbots.
- Rabe Abdalkareem, Emad Shihab, and Juergen Rilling. 2017. On Code Reuse from StackOverflow. Information Software Technology 88, C (Aug. 2017), 148--158. https://doi.org/10.1016/j.infsof.2017.04.005Google Scholar
- R. Abdalkareem, E. Shihab, and J. Rilling. 2017. What Do Developers Use the Crowd For? A Study Using Stack Overflow. IEEE Software 34, 2 (Mar 2017), 53--60. https://doi.org/10.1109/MS.2017.31Google ScholarDigital Library
- Ahmad Abdellatif, Khaled Badran, and Emad Shihab. 2020. MSRBot: Using Bots to Answer Questions from Software Repositories. Empirical Software Engineering (EMSE) (2020). https://doi.org/10.1007/s10664-019-09788-5Google Scholar
- E. Aghajani, G. Bavota, M. Linares-Vásquez, and M. Lanza. 2018. Automated Documentation of Android Apps. IEEE Transactions on Software Engineering (2018), 1--1. https://doi.org/10.1109/TSE.2018.2890652Google Scholar
- H. Agus Santoso, N. Anisa Sri Winarsih, E. Mulyanto, G. Wilujeng saraswati, S. Enggar Sukmana, S. Rustad, M. Syaifur Rohman, A. Nugraha, and F. Firdausillah. 2018. Dinus Intelligent Assistance (DINA) Chatbot for University Admission Services. In 2018 International Seminar on Application for Technology of Information and Communication. IEEE Press, 417--423.Google Scholar
- Syed Ahmed and Mehdi Bagherzadeh. 2018. What Do Concurrency Developers Ask About? A Large-scale Study Using Stack Overflow. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM '18). ACM, New York, NY, USA, Article 30, 10 pages. https://doi.org/10.1145/3239235.3239524Google ScholarDigital Library
- Amazon. 2019. Amazon Lex - Build Conversation Bots. https://aws.amazon.com/lex/. (Dec 2019). (Accessed on 12/12/2019).Google Scholar
- Apple. 2020. Siri - Apple. https://www.apple.com/ca/siri/. (2020). (Accessed on 01/08/2020).Google Scholar
- Amazon AWS. 2019. Document History for Amazon Lex - Amazon Lex. https://docs.aws.amazon.com/lex/latest/dg/doc-history.html. (2019). (Accessed on 12/12/2019).Google Scholar
- C.J. Baby, F. A. Khan, and J. N. Swathi. 2017. Home automation using IoT and a chatbot using natural language processing. In 2017 Innovations in Power and Advanced Computing Technologies (i-PACT). IEEE Press, 1--6. https://doi.org/10.1109/IPACT.2017.8245185Google Scholar
- Mehdi Bagherzadeh and Raffi Khatchadourian. 2019. Going Big: A Large-Scale Study on What Big Data Developers Ask. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 432--442. https://doi.org/10.1145/3338906.3338939Google ScholarDigital Library
- Kartik Bajaj, Karthik Pattabiraman, and Ali Mesbah. 2014. Mining Questions Asked by Web Developers. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). ACM, New York, NY, USA, 112--121. https://doi.org/10.1145/2597073.2597083Google ScholarDigital Library
- Anton Barua, Stephen W. Thomas, and Ahmed E. Hassan. 2014. What are developers talking about? An analysis of topics and trends in Stack Overflow. Empirical Software Engineering 19, 3 (01 Jun 2014), 619--654.Google Scholar
- David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. The Journal of Machine Learning Research 3 (March 2003), 993--1022. http://dl.acm.org/citation.cfm?id=944919.944937Google ScholarDigital Library
- Marion Boiteux. 2018. Messenger at F8 2018 - Messenger Developer Blog. https://blog.messengerdevelopers.com/messenger-at-f8-2018-44010dc9d2ea. (2018). (Accessed on 12/21/2019).Google Scholar
- Nick C. Bradley, Thomas Fritz, and Reid Holmes. 2018. Context-Aware Conversational Developer Assistants. In Proceedings of the 40th International Conference on Software Engineering (ICSE '18). Association for Computing Machinery, New York, NY, USA, 993--1003. https://doi.org/10.1145/3180155.3180238Google ScholarDigital Library
- Gillian Cameron, David Cameron, Gavin Megaw, Raymond Bond, Maurice Mulvenna, Siobhan O'Neill, Cherie Armour, and Michael McTear. 2018. Best Practices for Designing Chatbots in Mental Healthcare: A Case Study on IHelpr. In Proceedings of the 32nd International BCS Human Computer Interaction Conference (HCI '18). BCS Learning & Development Ltd., Swindon, GBR, Article Article 129, 5 pages. https://doi.org/10.14236/ewic/HCI2018.129Google ScholarDigital Library
- PACT Care. 2020. Florence - Your health assistant. https://www.florence.chat/. (2020). (Accessed on 01/08/2020).Google Scholar
- Chatbot application Life cycle 2019. Chatbot application Life cycle - Data Driven Investor - Medium. https://medium.com/datadriveninvestor/chatbot-application-life-cycle-8b2d083650a8. (June 2019). (Accessed on 12/16/2019).Google Scholar
- G. Chen, C. Chen, Z. Xing, and B. Xu. 2016. Learning a dual-language vector space for domain-specific cross-lingual question retrieval. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). 744--755.Google Scholar
- Google Dialogflow. 2020. Build an agent. https://cloud.google.com/dialogflow/docs/quick/build-agent. (2020). (Accessed on 01/16/2020).Google Scholar
- Saurabh "Dutta, Ger Joyce, and Jay" Brewer. "2018". "Utilizing Chatbots to Increase the Efficacy of Information Security Practitioners". In "Advances in Human Factors in Cybersecurity", Denise" "Nicholson (Ed.). "Springer International Publishing", "237-243".Google Scholar
- Stack Exchange. 2019. Stack Exchange Data Dump. https://archive.org/details/stackexchange. (2019). (Sept. 2019).Google Scholar
- Facebook. 2019. Wit.ai. https://wit.ai/. (2019). (Accessed on 12/12/2019).Google Scholar
- J. Gao, J. Chen, S. Zhang, X. He, and S. Lin. 2019. Recognizing Biomedical Named Entities by Integrating Domain Contextual Relevance Measurement and Active Learning. In 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). 1495--1499. https://doi.org/10.1109/ITNEC.2019.8728991Google Scholar
- Gensim. 2019. gensim: Topic modelling for humans. https://radimrehurek.com/gensim/. (2019). (Accessed on 12/03/2019).Google Scholar
- Google. 2020. Dialogflow. https://dialogflow.com/. (2020). (Accessed on 01/16/2020).Google Scholar
- Google. 2020. Google Assistant, your own personal Google. https://assistant.google.com/. (2020). (Accessed on 01/08/2020).Google Scholar
- Google. 2020. Integrations-Dialogflow Documentation. https://cloud.google.com/dialogflow/docs/integrations/. (2020). (Accessed on 01/16/2020).Google Scholar
- Junxiao Han, Emad Shihab, Zhiyuan Wan, Shuiguang Den, and Xin Xia. 2019. What do Programmers Discuss about Deep Learning Frameworks. Empirical Software Engineering(EMSE) (2019), To Appear.Google Scholar
- Z. Jin, K. Y. Chee, and X. Xia. 2019. What Do Developers Discuss about Biometric APIs?. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). 348--352. https://doi.org/10.1109/ICSME.2019.00053Google ScholarCross Ref
- C. Lebeuf, M. Storey, and A. Zagalsky. 2018. Software Bots. IEEE Software 35, 1 (January 2018), 18--23. https://doi.org/10.1109/MS.2017.4541027Google ScholarCross Ref
- C. Lebeuf, A. Zagalsky, M. Foucault, and M. Storey. 2019. Defining and Classifying Software Bots: A Faceted Taxonomy. In 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE). 1--6. https://doi.org/10.1109/BotSE.2019.00008Google Scholar
- Carlene R Lebeuf. 2018. A taxonomy of software bots: towards a deeper understanding of software bot characteristics. Ph.D. Dissertation.Google Scholar
- Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. http://mallet.cs.umass.edu/. (2002). (Accessed on 12/03/2019).Google Scholar
- Mary McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia medica 22 (10 2012), 276--82. https://doi.org/10.11613/BM.2012.031Google Scholar
- Microsoft. 2019. LUIS (Language Understanding) - Cognitive Services - Microsoft Azure. https://www.luis.ai/home. (2019). (Accessed on 12/12/2019).Google Scholar
- Microsoft. 2020. Microsoft Bot Framework. https://dev.botframework.com/. (2020). (Accessed on 01/16/2020).Google Scholar
- Microsoft. 2020. Quickstart: Create a new app in the LUIS portal - Azure Cognitive Services | Microsoft Docs. https://docs.microsoft.com/en-us/azure/cognitive-services/luis/get-started-portal-build-app. (2020). (Accessed on 01/16/2020).Google Scholar
- Milja Milenkovic. 2019. The Future Is Now- 37 Fascinating Chatbot Statistics. https://www.smallbizgenius.net/by-the-numbers/chatbot-statistics/. (Oct. 2019). (Accessed on 12/18/2019).Google Scholar
- Sarah Nadi, Stefan Krüger, Mira Mezini, and Eric Bodden. 2016. Jumping Through Hoops: Why Do Java Developers Struggle with Cryptography APIs?. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). ACM, New York, NY, USA, 935--946. https://doi.org/10.1145/2884781.2884790Google ScholarDigital Library
- Natural Language Toolkit (NLTK). 2019. Natural Language Toolkit - NLTK 3.4.5 documentation. https://www.nltk.org/. (2019). (Accessed on 12/12/2019).Google Scholar
- Natural Language Toolkit (NLTK). 2019. NLTK's list of english stopwords. https://gist.github.com/sebleier/554280. (2019). (Accessed on 12/23/2019).Google Scholar
- Stack Overflow. 2017. node.js - How to start a conversation from nodejs client to microsoft bot - Stack Overflow. https://stackoverflow.com/questions/46183295/how-to-start-a-conversation-from-nodejs-client-to-microsoft-bot. (2017). (Accessed on 12/20/2019).Google Scholar
- Stack Overflow. 2017. python - Dataset to train MITIE ner model - Stack Overflow. https://stackoverflow.com/questions/46602495/dataset-to-train-mitie-ner-model. (2017). (Accessed on 01/13/2020).Google Scholar
- Stack Overflow. 2017. Rasa nlu parse request giving wrong intent result - Stack Overflow. https://stackoverflow.com/questions/46466222/rasa-nlu-parse-request-giving-wrong-intent-result. (2017). (Accessed on 12/20/2019).Google Scholar
- Stack Overflow. 2018. Facebook Chat bot (PHP webhook) sending multiple replies - Stack Overflow. https://stackoverflow.com/questions/36609549/facebook-chat-bot-php-webhook-sending-multiple-replies. (2018). (Accessed on 12/20/2019).Google Scholar
- Stack Overflow. 2019. botframework - How to add custom choices displayed through Prompt options inside Cards & trigger actions on choice click in BOT V4 using c#? - Stack Overflow. (2019). https://stackoverflow.com/questions/56280689/how-to-add-custom-choices-displayed-through-prompt-options-inside-cards-trigge(Accessed on 01/16/2020).Google Scholar
- Stack Overflow. 2019. nlp - Is there a dataset that provides shopping conversations? - Stack Overflow. https://stackoverflow.com/questions/55324833/is-there-a-dataset-that-provides-shopping-conversations. (2019). (Accessed on 01/13/2020).Google Scholar
- Stack Overflow. 2019. node.js - How to store and retrieve the chat history of the dialogflow? - Stack Overflow. https://stackoverflow.com/questions/49665510/how-to-store-and-retrieve-the-chat-history-of-the-dialogflow. (2019). (Accessed on 12/21/2019).Google Scholar
- Stack Overflow. 2019. python - How to resume or restart paused conversation in RASA - Stack Overflow. https://stackoverflow.com/questions/57365685/how-to-resume-or-restart-paused-conversation-in-rasa. (2019). (Accessed on 12/20/2019).Google Scholar
- Stack Overflow. 2019. Stack Overflow Developer Survey 2019. https://insights.stackoverflow.com/survey/2019. (2019). (Accessed on 01/09/2020).Google Scholar
- Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the Space of Topic Coherence Measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (WSDM '15). ACM, New York, NY, USA, 399--408. https://doi.org/10.1145/2684822.2685324Google ScholarDigital Library
- Christoffer Rosen and Emad Shihab. 2016. What Are Mobile Developers Asking About? A Large Scale Study Using Stack Overflow. Empirical Software Engineering 21, 3 (June 2016), 1192--1223. https://doi.org/10.1007/s10664-015-9379-3Google ScholarDigital Library
- B. Rychalska, H. Glabska, and A. Wroblewska. 2018. Multi-Intent Hierarchical Natural Language Understanding for Chatbots. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS). 256--259. https://doi.org/10.1109/SNAMS.2018.8554770Google Scholar
- A. Abdellatif, D. Costa, K. Badran, R. Abdalkareem, E. Shihab. 2020. Dataset. https://zenodo.org/record/3610714. (2020). (Accessed on 01/16/2020).Google Scholar
- Spearman. 2008. Spearman Rank Correlation Coefficient. Springer New York, New York, NY, 502--505. https://doi.org/10.1007/978-0-387-32833-1_379Google Scholar
- Margaret-Anne Storey and Alexey Zagalsky. 2016. Disrupting Developer Productivity One Bot at a Time. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2016). ACM, New York, NY, USA, 928--931. https://doi.org/10.1145/2950290.2983989Google ScholarDigital Library
- Sumo. 2020. 5 Ecommerce Chatbots (Plus How To Build Your Own In 15 Minutes). https://sumo.com/stories/ecommerce-chatbot-marketing. (2020). (Accessed on 01/08/2020).Google Scholar
- TechCrunch. 2017. Wit.ai is shutting down Bot Engine as Facebook rolls NLP into its updated Messenger Platform. (2017). https://techcrunch.com/2017/07/27/wit-ai-is-shutting-down-bot-engine-as-facebook-rolls-nlp-into-its-updated-messenger-platform (Accessed on 12/12/2019).Google Scholar
- Y. Tian, F. Thung, A. Sharma, and D. Lo. 2017. APIBot: Question answering bot for API documentation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 153--158. https://doi.org/10.1109/ASE.2017.8115628Google ScholarCross Ref
- Carlos Toxtli, Andrés Monroy-Hernández, and Justin Cranshaw. 2018. Understanding Chatbot-mediated Task Management. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 58, 6 pages. https://doi.org/10.1145/3173574.3173632Google ScholarDigital Library
- Christoph Treude, Ohad Barzilay, and Margaret-Anne Storey. 2011. How Do Programmers Ask and Answer Questions on the Web? (NIER Track). In Proceedings of the 33rd International Conference on Software Engineering (ICSE '11). ACM, New York, NY, USA, 804--807. https://doi.org/10.1145/1985793.1985907Google ScholarDigital Library
- Simon Urli, Zhongxing Yu, Lionel Seinturier, and Martin Monperrus. 2018. How to Design a Program Repair Bot? Insights from the Repairnator Project. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP '18). ACM, New York, NY, USA, 95--104. https://doi.org/10.1145/3183519.3183540Google ScholarDigital Library
- Stefano Valtolina, Barbara Rita Barricelli, and Serena Di Gaetano. 2020. Communicability of traditional interfaces VS chatbots in healthcare and smart home domains. Behaviour & Information Technology 39, 1 (2020), 108--132. https://doi.org/10.1080/0144929X.2019.1637025arXiv: https://doi.org/10.1080/0144929X.2019.1637025Google ScholarCross Ref
- P. K. Venkatesh, S. Wang, F. Zhang, Y. Zou, and A. E. Hassan. 2016. What Do Client Developers Concern When Using Web APIs? An Empirical Study on Developer Forums and Stack Overflow. In 2016 IEEE International Conference on Web Services (ICWS). 131--138. https://doi.org/10.1109/ICWS.2016.25Google Scholar
- Z. Wan, X. Xia, and A. E. Hassan. 2019. What is Discussed about Blockchain? A Case Study on the Use of Balanced LDA and the Reference Architecture of a Domain to Capture Online Discussions about Blockchain platforms across the Stack Exchange Communities. IEEE Transactions on Software Engineering (2019), 1--1. https://doi.org/10.1109/TSE.2019.2921343Google Scholar
- Joseph Weizenbaum. 1966. ELIZA-A Computer Program for the Study of Natural Language Communication between Man and Machine. Commun. ACM 9, 1 (Jan. 1966), 36--45. https://doi.org/10.1145/365153.365168Google ScholarDigital Library
- Marvin Wyrich and Justus Bogner. 2019. Towards an Autonomous Bot for Automatic Source Code Refactoring. In Proceedings of the 1st International Workshop on Bots in Software Engineering (BotSE '19). IEEE Press, Piscataway, NJ, USA, 24--28. https://doi.org/10.1109/BotSE.2019.00015Google ScholarDigital Library
- Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A New Chatbot for Customer Service on Social Media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, New York, NY, USA, 3506--3510. https://doi.org/10.1145/3025453.3025496Google ScholarDigital Library
- Bowen Xu, Zhenchang Xing, Xin Xia, and David Lo. 2017. AnswerBot: Automated Generation of Answer Summary to Developers' Technical Questions. In Proceedings of the 32Nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017). IEEE Press, Piscataway, NJ, USA, 706--716. http://dl.acm.org/citation.cfm?id=3155562.3155650Google ScholarDigital Library
- Xin-Li Yang, David Lo, Xin Xia, Zhi-Yuan Wan, and Jian-Ling" Sun. 2016. What Security Questions Do Developers Ask? A Large-Scale Study of Stack Overflow Posts. Journal of Computer Science and Technology 31, 5 (01 Sep 2016), 910--924.Google ScholarCross Ref
- D. Ye, Z. Xing, C. Y. Foo, Z. Q. Ang, J. Li, and N. Kapre. 2016. Software-Specific Named Entity Recognition in Software Engineering Social Content. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. 90--101. https://doi.org/10.1109/SANER.2016.10Google Scholar
- Shayan Zamanirad, Boualem Benatallah, Moshe Chai Barukh, Fabio Casati, and Carlos Rodriguez. 2017. Programming Bots by Synthesizing Natural Language Expressions into API Invocations. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017). IEEE Press, 832--837.Google ScholarDigital Library
- H. Zhang, S. Wang, T. Chen, and A. E. Hassan. 2019. Reading Answers on Stack Overflow: Not Enough! IEEE Transactions on Software Engineering (2019), 1--1. https://doi.org/10.1109/TSE.2019.2954319Google Scholar
- Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive Co-attention Network for Named Entity Recognition in Tweets. (2018). https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16432Google Scholar
Index Terms
- Challenges in Chatbot Development: A Study of Stack Overflow Posts
Index terms have been assigned to the content through auto-classification.
Recommendations
Chatbot with Touch and Graphics: An Interaction of Users for Emotional Expression and Turn-taking
CUI '20: Proceedings of the 2nd Conference on Conversational User InterfacesUse of chatbots for emotional exchange is recently increasing in various domains. However, as existing chatbots have been considered in terms of natural language processing techniques for interaction with text-based chatting, chatbot interaction with ...
Chatbot or Chat-Blocker: Predicting Chatbot Popularity before Deployment
DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems ConferenceChatbots are widely employed in various scenarios. However, given the high costs of chatbot development and chatbots’ tremendous social influence, chatbot failures may inevitably lead to a huge economic loss. Previous chatbot evaluation frameworks rely ...
Chatbot learning partners: Connecting learning experiences, interest and competence
AbstractConversation practice, while paramount for all language learners, can be difficult to get enough of and very expensive. In this mobile age, chatbots are an obvious means of filling this gap, but have yet to realize their potential as ...
Highlights- Students' experiences with chatbots as language learning partners was assessed.
Comments