Skip to main content
SpringerLink
Log in

Usability and User Experience of a Chat Application with Integrated Educational Chatbot Functionalities

  • Conference paper
  • First Online:
Learning and Collaboration Technologies: Games and Virtual Environments for Learning (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12785))

Included in the following conference series:

Abstract

In recent years chatbots have found their use in education, especially in higher education (HE), but are they really serving a purpose and helping both students and faculty staff? Since bad experiences can discourage potential users, chatbots have to be carefully designed to be useful and to offer the best possible experience to its users. To explore this premise, in this paper we will tackle one aspect of chatbot design through evaluation of usability and user experience of a chat application Differ, which has integrated chatbot functionalities. For the duration of one semester at the Faculty of Organization and Informatics’ students used Differ on several courses. Differ was introduced as a tool to facilitate communication related to courses and students’ obligations, between students and educators by initiating communication through a Differ’s chatbot called Buddy Bot. Differ was also seen as a tool to enhance students’ connections and social life through creation of private student communities. After the end of semester, students’ opinions about Differ were evaluated through means of UEQ and SUS questionnaires which are frequently used for user experience and usability evaluation. Results of the study showed that Differ’s perceived usability and user experience are slightly above average, but with a lot of room for improvements, especially in the context of novelty, stimulation and chatbot functionalities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Stapić, Z., Horvat, A., Plantak Vukovac, D.: Designing a faculty Chatbot through user-centered design approach. In: Stephanidis, C., et al. (eds.) HCII 2020. LNCS, vol. 12425, pp. 472–484. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60128-7_36

    Chapter  Google Scholar 

  2. Pérez, J.Q., Daradoumis, T., Puig, J.M.M.: Rediscovering the use of chatbots in education: a systematic literature review. Comput. Appl. Eng. Educ. 28, 1549–1565 (2020). https://doi.org/10.1002/cae.22326

    Article  Google Scholar 

  3. Brandtzaeg, P.B., Følstad, A.: Chatbots: changing user needs and motivations. Interact. 25, 38–43 (2018). https://doi.org/10.1145/3236669

    Article  Google Scholar 

  4. Tsivitanidou, O., Ioannou, A.: Users’ needs assessment for chatbots’ use in higher education. In: Strahonja, V., Hertweck, D., Kirinić, V. (eds.) Proceeding of 31st Central European Conference on Information and Intelligent Systems, p. 8. University of Zagreb, Faculty of Organization and Informatics, Varaždin (2020).

    Google Scholar 

  5. Schrepp, M., Hinderks, A., Thomaschewski, J.: Construction of a benchmark for the user experience questionnaire (UEQ). Int. J. Interact. Multimed. Artif. Intell. 4, 40 (2017). https://doi.org/10.9781/ijimai.2017.445

    Article  Google Scholar 

  6. Følstad, A., Brandtzaeg, P.B.: Users’ experiences with chatbots: findings from a questionnaire study. Qual. User Experience 5(1), 1–14 (2020). https://doi.org/10.1007/s41233-020-00033-2

    Article  Google Scholar 

  7. New DigitasLBi research shows more than 1 in 3 Americans are willing to make purchases via Chatbots. https://www.digitas.com/en-us/news/press-releases/new-digitaslbi-research-shows-more-than-1-in-3-americans-are-willing-to-make-purchases-via-chatbots. Accessed 12 Jan 2021

  8. Hassenzahl, M., Tractinsky, N.: User experience - a research agenda. Behav. Inf. Technol. 25, 91–97 (2006). https://doi.org/10.1080/01449290500330331

    Article  Google Scholar 

  9. Caro-Alvaro, S., Garcia-Lopez, E., Garcia-Cabot, A., de-Marcos, L., Martinez-Herraiz, J.-J.: Identifying usability issues in instant messaging apps on iOS and android platforms. Mob. Inf. Syst. 2018(1), 1–19 (2018). https://doi.org/10.1155/2018/2056290

  10. Darin, T., Andrade, R., Macedo, J., Araújo, D., Mesquita, L., Sánchez, J.: Usability and UX evaluation of a mobile social application to increase students-faculty interactions. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 618, pp. 21–29. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40542-1_4

    Chapter  Google Scholar 

  11. Dibitonto, M., Leszczynska, K., Tazzi, F., Medaglia, C.M.: Chatbot in a campus environment: design of LiSA, a virtual assistant to help students in their university life. In: Kurosu, M. (ed.) HCI 2018. LNCS, vol. 10903, pp. 103–116. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91250-9_9

    Chapter  Google Scholar 

  12. Hien, H.T., Cuong, P.-N., Nam, L.N.H., Nhung, H.L.T.K., Thang, L.D.: Intelligent assistants in higher-education environments: the FIT-EBot, a Chatbot for administrative and learning support. In: Proceedings of the Ninth International Symposium on Information and Communication Technology, pp. 69–76. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3287921.3287937

  13. Zahour, O., Benlahmar, E.H., Eddaoui, A., Ouchra, H., Hourrane, O.: A system for educational and vocational guidance in morocco: Chatbot E-orientation. Procedia Comput. Sci. 175, 554–559 (2020). https://doi.org/10.1016/j.procs.2020.07.079

    Article  Google Scholar 

  14. Sandu, N., Gide, E.: Adoption of AI-Chatbots to enhance student learning experience in higher education in India. In: 2019 18th International Conference on Information Technology Based Higher Education and Training (ITHET), pp. 1–5. IEEE, Magdeburg (2019). https://doi.org/10.1109/ITHET46829.2019.8937382

  15. Benotti, L., Martínez, M.C., Schapachnik, F.: Engaging high school students using chatbots. In: Proceedings of the 2014 conference on Innovation and technology in computer science education, pp. 63–68. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2591708.2591728

  16. Liu, R., Dong, Z.: A study of user experience in knowledge-based QA Chatbot design. In: Karwowski, W., Ahram, T. (eds.) IHSI 2019. AISC, vol. 903, pp. 589–593. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11051-2_89

    Chapter  Google Scholar 

  17. Dahiya, M.: A tool of conversation: Chatbot. Int. J. Comput. Sci. Eng. 4 (2017)

    Google Scholar 

  18. Maroengsit, W., Piyakulpinyo, T., Phonyiam, K., Pongnumkul, S., Chaovalit, P., Theeramunkong, T.: A survey on evaluation methods for Chatbots. In: Proceedings of the 2019 7th International Conference on Information and Education Technology, pp. 111–119. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3323771.3323824

  19. Casas, J., Tricot, M.-O., Abou Khaled, O., Mugellini, E., Cudré-Mauroux, P.: Trends and methods in Chatbot evaluation. In: Companion Publication of the 2020 International Conference on Multimodal Interaction, pp. 280–286. ACM, Virtual Event Netherlands (2020). https://doi.org/10.1145/3395035.3425319

  20. Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., Mctear, M.: Usability testing of a healthcare Chatbot: can we use conventional methods to assess conversational user interfaces? In: Proceedings of the 31st European Conference on Cognitive Ergonomics, pp. 207–214. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3335082.3335094

  21. te Pas, M.E., Rutten, W.G.M.M., Bouwman, R.A., Buise, M.P.: User experience of a Chatbot questionnaire versus a regular computer questionnaire: prospective comparative study. JMIR Med. Inform. 8, e21982 (2020). https://doi.org/10.2196/21982

    Article  Google Scholar 

  22. Differ - Helping you connect with other new students. http://www.differ.chat. Accessed 22 Feb 2021

  23. Differ for educators - improve your students wellbeing and success. http://www.differ.chat. Accessed 22 Feb 2021

  24. Studente, S., Ellis, S., Garivaldis, S.F.: Exploring the potential of Chatbots in higher education: a preliminary study. 14, 4 (2020)

    Google Scholar 

  25. Čižmešija, A., Horvat, A., Vukovac, D.P.: Improving student engagement and course completion using Chatbot application. In: INTED2021 Proceedings (in press), p. 10. IATED, Spain (2021). https://doi.org/10.21125/inted.2021

  26. Forster, Y., Hergeth, S., Naujoks, F., Krems, J.F.: How usability can save the day - methodological considerations for making automated driving a success story. In: Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 278–290. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3239060.3239076

  27. Díaz-Oreiro, L.: Quesada, guerrero: standardized questionnaires for user experience evaluation: a systematic literature review. Proceedings 31, 14 (2019). https://doi.org/10.3390/proceedings2019031014

    Article  Google Scholar 

  28. Kortum, P., Sorber, M.: Measuring the usability of mobile applications for phones and tablets. Int. J. Hum.-Comput. Interact. 31, 518–529 (2015). https://doi.org/10.1080/10447318.2015.1064658

    Article  Google Scholar 

  29. Kortum, P.T., Bangor, A.: Usability ratings for everyday products measured with the system usability scale. Int. J. Hum.-Comput. Interact. 29, 67–76 (2013). https://doi.org/10.1080/10447318.2012.681221

    Article  Google Scholar 

  30. Harrati, N., Bouchrika, I., Tari, A., Ladjailia, A.: Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Comput. Hum. Behav. 61, 463–471 (2016). https://doi.org/10.1016/j.chb.2016.03.051

    Article  Google Scholar 

  31. Laugwitz, B., Held, T., Schrepp, M.: Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (ed.) USAB 2008. LNCS, vol. 5298, pp. 63–76. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89350-9_6

    Chapter  Google Scholar 

  32. User Experience Questionnaire (UEQ). https://www.ueq-online.org/. Accessed 12 Jan 2021

  33. Sauro, J.: Measuring usability with the System Usability Scale (SUS). https://measuringu.com/sus/. Accessed 11 Jan 2021

  34. Sauro, J.: MeasuringU: 5 Ways to Interpret a SUS Score. https://measuringu.com/interpret-sus-score/. Accessed 14 Feb 2021

Download references

Acknowledgements

This work is a part of the project “Best practices of pedagogical chatbots in higher education / EDUBOTS” which is funded under the Erasmus+ KA2: Cooperation for innovation and the exchange of good practices – Knowledge Alliances (grant agreement no: 612446, project ref.: 612446-EPP-1–2019-1-NO-EPPKA2-KA).

Author information

Authors and Affiliations

  1. Faculty of Organization and Informatics, University of Zagreb, Pavlinska 2, 42000, Varaždin, Croatia

    Dijana Plantak Vukovac, Ana Horvat & Antonela Čižmešija

Authors
  1. Dijana Plantak Vukovac
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ana Horvat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonela Čižmešija
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dijana Plantak Vukovac .

Editor information

Editors and Affiliations

  1. Department of Multimedia and Graphic Arts, Cyprus University of Technology, Lemesos, Cyprus

    Panayiotis Zaphiris

  2. Research Center on Interactive Media, Smart Systems and Emerging Technologies (CYENS), Cyprus University of Technology, Limassol, Cyprus

    Andri Ioannou

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Plantak Vukovac, D., Horvat, A., Čižmešija, A. (2021). Usability and User Experience of a Chat Application with Integrated Educational Chatbot Functionalities. In: Zaphiris, P., Ioannou, A. (eds) Learning and Collaboration Technologies: Games and Virtual Environments for Learning. HCII 2021. Lecture Notes in Computer Science(), vol 12785. Springer, Cham. https://doi.org/10.1007/978-3-030-77943-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77943-6_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77942-9

  • Online ISBN: 978-3-030-77943-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation