Skip to main content
SpringerLink
Log in

Methods for Evaluating Conversational Agents’ Communicability, Acceptability and Accessibility Degree

  • Conference paper
  • First Online:
Human-Computer Interaction – INTERACT 2023 (INTERACT 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14143))

Included in the following conference series:

  • 650 Accesses

Abstract

Recently, there has been an increasing interest in language-based interactions with technology. Driven by the success of intelligent personal assistants, the number of conversation-based interactions is growing in several domains. Nevertheless, the literature highlights a lack of models specifically studied to analyse communication blocks and the level of accessibility and acceptability by the user that can characterise Human-Agent dialogue. This paper aims to learn how much an agent can be accessible, how much the communication is understandable, and if it brings to a successful conclusion by extending two known models, the UTAUT2 and CEM. For both models, we defined new indicators to analyse communicability, acceptability degrees and accessibility level using WCAG guidelines. We designed two conversational agents to test our models and conducted preliminary tests. A first agent is used to assist students in following remote digital courses, and a second to help older people in their daily activities and to monitor the indexes of active life defined to control the trend of older people’s physical-cognitive state.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.whoteach.it/ Last access: 2023–06-08.

References

  1. Boonstra, L.: Definitive Guide to Conversational AI with Dialogflow and Google Cloud. Apress (2021)

    Google Scholar 

  2. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology: toward a unified view. MIS Q., 425–478 (2003)

    Google Scholar 

  3. Venkatesh, V., Davis, F.D.: A theoretical extension of the technology acceptance model: four longitudinal field studies. Manag. Sci. 46(2), 186–204 (2000)

    Article  Google Scholar 

  4. Warshaw, P.R.: A new model for predicting behavioral intentions: an alternative to Fishbein. J. Mark. Res. 17(2), 153–172 (1980)

    Article  Google Scholar 

  5. Mattos, B.A.M., Prates, R.O.: An overview of the communicability evaluation method for collaborative systems. In: IADIS International Conference WWW/Internet 2011, Rio de Janeiro. Proceedings of WWW/Internet 2011, pp. 129–136 (2011)

    Google Scholar 

  6. De Souza, C.S., Leitão, C.F.: Semiotic Engineering Methods for Scientific Research in HCI. Morgan and Claypool Publishers, San Rafael (2009)

    Google Scholar 

  7. Prates, R.O., de Souza, C.S., Barbosa, S.D.J.: A method for evaluating the communicability of user interfaces. ACM Interact. 7(1), 31–38 (2000)

    Article  Google Scholar 

  8. Okonkwo, C.W., Ade-Ibijola, A.: Chatbots applications in education: a systematic review. Comput. Educ. Artif. Intell. 2, 100033 (2021)

    Article  Google Scholar 

  9. WCAG 3.0: https://www.w3.org/TR/wcag-3.0/ and WCAG 2.1: https://wcag.it/. Accessed 24 Apr 2023

  10. Medeiros, R.P., Ramalho, G.L., Falcão, T.P.: A systematic literature review on teaching and learning introductory programming in higher education. IEEE Trans. Educ. 62(2), 77–90 (2018)

    Article  Google Scholar 

  11. Smutny, P., Schreiberova, P.: Chatbots for learning: a review of educational chatbots for the Facebook Messenger. Comput. Educ. 151, 103862 (2020)

    Article  Google Scholar 

  12. Alias, S., Sainin, M.S., Soo Fun, T., Daut, N.: Identification of conversational intent pattern using pattern-growth technique for academic chatbot. In: Chamchong, R., Wong, K.W. (eds.) MIWAI 2019. LNCS (LNAI), vol. 11909, pp. 263–270. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33709-4_24

    Chapter  Google Scholar 

  13. Wu, E.H.K., Lin, C.H., Ou, Y.Y., Liu, C.Z., Wang, W.K., Chao, C.Y.: Advantages and constraints of a hybrid model K-12 E-learning assistant chatbot. IEEE Access 8, 77788–77801 (2020)

    Article  Google Scholar 

  14. Murad, D.F., Irsan, M., Akhirianto, P.M., Fernando, E., Murad, S.A., Wijaya, M.H.: Learning support system using chatbot in “Kejar C Package” homeschooling program. In: 2019 International Conference on Information and Communications Technology (ICOIACT), pp. 32–37. IEEE, July 2019

    Google Scholar 

  15. Lam, C.S.N., Chan, L.K., See, C.Y.H.: Converse, connect and consolidate–the development of an artificial intelligence chatbot for health sciences education. In: Frontiers in Medical and Health Sciences Education Conference. Bau Institute of Medical and Health Sciences Education, Li Ka Shing Faculty of Medicine, The University of Hong Kong (2018)

    Google Scholar 

  16. El Hefny, W., El Bolock, A., Herbert, C., Abdennadher, S.: Applying the character-based chatbots generation framework in education and healthcare. In: Proceedings of the 9th International Conference on Human-Agent Interaction, pp. 121–129, November 2021

    Google Scholar 

  17. Sreelakshmi, A.S., Abhinaya, S.B., Nair, A., Nirmala, S.J.: A question answering and quiz generation chatbot for education. In: 2019 Grace Hopper Celebration India (GHCI), pp. 1–6. IEEE, November 2019

    Google Scholar 

  18. https://rasa.com/. Accessed 24 Apr 2023

  19. https://it.legacy.reactjs.org/. Accessed 24 Apr 2023

  20. https://cloud.google.com/dialogflow?hl=it. Accessed 24 Apr 2023

  21. https://docs.flutter.dev/. Accessed 24 Apr 2023

  22. Chaves, A.P., Gerosa, M.A.: How should my chatbot interact? A survey on social characteristics in human–chatbot interaction design. Int. J. Hum. Comput. Interact., 1–30 (2020)

    Google Scholar 

  23. Rapp, A., Curti, L., Boldi, A.: The human side of human-chatbot interaction: a systematic literature review of ten years of research on text-based chatbots. Int. J. Hum. Comput. Stud., 102630 (2021)

    Google Scholar 

  24. De Veer, A.J., Peeters, J.M., Brabers, A.E., Schellevis, F.G., Rademakers, J.J.J., Francke, A.L.: Determinants of the intention to use e-health by community dwelling older people. BMC Health Serv. Res. 15(1), 1–9 (2015)

    Google Scholar 

  25. Liu, C.F., Tsai, Y.C., Jang, F.L.: Patients’ acceptance towards a web-based personal health record system: an empirical study in Taiwan. Int. J. Environ. Res. Public Health 10(10), 5191–5208 (2013)

    Article  Google Scholar 

  26. Kohnke, A., Cole, M.L., Bush, R.: Incorporating UTAUT predictors for understanding home care patients’ and clinician’s acceptance of healthcare telemedicine equipment. J. Technol. Manag. Innov. 9(2), 29–41 (2014)

    Article  Google Scholar 

  27. Cimperman, M., Brenčič, M.M., Trkman, P.: Analysing older users’ home telehealth services acceptance behavior—applying an extended UTAUT model. Int. J. Med. Inform. 90, 22–31 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Università degli Studi di Milano, Milano, Italy

    Stefano Valtolina

  2. Social Things SRL, Milano, Italy

    Ricardo Anibal Matamoros & Francesco Epifania

Authors
  1. Stefano Valtolina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ricardo Anibal Matamoros
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francesco Epifania
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefano Valtolina .

Editor information

Editors and Affiliations

  1. University of West London, London, UK

    José Abdelnour Nocera

  2. Reykjavik University, Reykjavik, Iceland

    Marta Kristín Lárusdóttir

  3. University of York, York, UK

    Helen Petrie

  4. University of Bari Aldo Moro, Bari, Italy

    Antonio Piccinno

  5. Université Côte d’Azur, Sophia Antipolis Cedex, France

    Marco Winckler

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Valtolina, S., Matamoros, R.A., Epifania, F. (2023). Methods for Evaluating Conversational Agents’ Communicability, Acceptability and Accessibility Degree. In: Abdelnour Nocera, J., Kristín Lárusdóttir, M., Petrie, H., Piccinno, A., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2023. INTERACT 2023. Lecture Notes in Computer Science, vol 14143. Springer, Cham. https://doi.org/10.1007/978-3-031-42283-6_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42283-6_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42282-9

  • Online ISBN: 978-3-031-42283-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation