Skip to main content
SpringerLink
Log in

A Framework for Humanization Evaluation in Chatbots

  • Conference paper
  • First Online:
Human-Computer Interaction (HCII 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14013))

Included in the following conference series:

  • 953 Accesses

Abstract

Chatbots play a central role in modern software systems regarding customer services. They have the potential of applicability for different purposes. A key open research challenge is how humanizing and evaluating chatbots during their development. Humanized chatbots carry out a fluid and pleasant conversation with the user, demonstrating empathy and personality. In this article, we aim to study and develop an evaluation method that indicates the level of humanization of a chatbot under analysis. Our framework comprises two objectives and questionnaires, which are applied to establish an assessment adaptable to the different objectives of using chatbots (commercial or therapeutic, for example). We combine these questionnaires to generate evaluation metrics, which provide a humanization score for the evaluated software. Our method helps designers identify specific factors that affect users’ experience interacting with chatbots. We carry out a case study on the application of our framework and reveal the main findings about its applicability.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The author of this article acted as a monitor in the course by assisting in applying the proposed methodology.

References

  1. Adam, M., Wessel, M., Benlian, A., et al.: AI-based chatbots in customer service and their effects on user compliance. Electron. Mark. 9(2), 204 (2020)

    Google Scholar 

  2. Adamopoulou, E., Moussiades, L.: An overview of chatbot technology. In: Maglogiannis, I., Iliadis, L., Pimenidis, E. (eds.) AIAI 2020. IAICT, vol. 584, pp. 373–383. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49186-4_31

    Chapter  Google Scholar 

  3. Alzorba, S., Günther, C., Popovici, N., Tammer, C.: A new algorithm for solving planar multiobjective location problems involving the Manhattan norm. Eur. J. Oper. Res. 258(1), 35–46 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  4. Balaji, D.: Assessing user satisfaction with information chatbots: a preliminary investigation. Master’s thesis, University of Twente (2019)

    Google Scholar 

  5. Bansal, H., Khan, R.: A review paper on human computer interaction. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 8, 53–56 (2018)

    Article  Google Scholar 

  6. Croasmun, J.T., Ostrom, L.: Using likert-type scales in the social sciences. J. Adult Educ. 40(1), 19–22 (2011)

    Google Scholar 

  7. Edmondson, D.: Likert scales: a history. In: Proceedings of the Conference on Historical Analysis and Research in Marketing, vol. 12, pp. 127–133 (2005)

    Google Scholar 

  8. Følstad, A., Brandtzaeg, P.B.: Users’ experiences with chatbots: findings from a questionnaire study. Quality User Exper. 5(1), 1–14 (2020). https://doi.org/10.1007/s41233-020-00033-2

    Article  Google Scholar 

  9. Gelici, A.A.: Change in digital business: chatbots and their significance on online purchasing, B. S. thesis, University of Twente (2020)

    Google Scholar 

  10. Go, E., Sundar, S.S.: Humanizing chatbots: the effects of visual, identity and conversational cues on humanness perceptions. Comput. Hum. Behav. 97, 304–316 (2019)

    Article  Google Scholar 

  11. Haslam, N., Loughnan, S., Holland, E.: The psychology of humanness. Objectification and (de) Humanization, pp. 25–51 (2013)

    Google Scholar 

  12. Joshi, A., Kale, S., Chandel, S., Pal, D.K.: Likert scale: explored and explained. British J. Appl. Sci. Technol. 7(4), 396 (2015)

    Article  Google Scholar 

  13. Kar, R., Haldar, R.: Applying chatbots to the internet of things: opportunities and architectural elements. arXiv preprint arXiv:1611.03799 (2016)

  14. Khanna, A., Pandey, B., Vashishta, K., Kalia, K., Pradeepkumar, B., Das, T.: A study of today’s A.I. through chatbots and rediscovery of machine intelligence. Int. J. u- e-Serv. Sci. Technol. 8(7), 277–284 (2015)

    Article  Google Scholar 

  15. Kříž, J.: Chatbot for laundry and dry cleaning service. Masaryk University (2017)

    Google Scholar 

  16. McLeod, S.: Likert scale. https://www.simplypsychology.org/ (2014)

  17. Mills, G.E., Gay, L.R.: Educational research: competencies for analysis and applications. ERIC (2019)

    Google Scholar 

  18. Nadkarni, P.M., Ohno-Machado, L., Chapman, W.W.: Natural language processing: an introduction. J. Am. Med. Inform. Assoc. 18(5), 544–551 (2011)

    Article  Google Scholar 

  19. Park, K.: Preventive and social medicine (2013)

    Google Scholar 

  20. Paschoal, L.N., Nogueira, L.L., Chicon, P.M.M.: Agentes conversacionais pedagógicos: Uma discussão inicial sobre conceitos, estratégias de desenvolvimento e oportunidades de pesquisa. Digitalização da Educação: Desafios e Estratégias para a Educação da Geração Conectada, p. 23 (2020)

    Google Scholar 

  21. Pasquali, L.: Psychometrics. Rev. Esc. Enferm. U.S.P. 43, 992–999 (2009)

    Article  Google Scholar 

  22. Peras, D.: Chatbot evaluation metrics. Economic and Social Development: book of Proceedings, pp. 89–97 (2018)

    Google Scholar 

  23. Ravinder, E.B., Saraswathi, D.A.: Literature review of cronbachalphacoefficient (a) and mcdonald’s omega coefficient (\(\omega \)). Eur. J. Molecular Clin. Med. 7(6), 2943–2949 (2020)

    Google Scholar 

  24. Revelle, W.: Hierarchical cluster analysis and the internal structure of tests. Multivar. Behav. Res. 14(1), 57–74 (1979)

    Article  Google Scholar 

  25. Viriato, P.J.d.S., Souza, R.R.d., Villas, L.A., dos Reis, J.C.: Revealing chatbot humanization impact factors. In: HCI International 2023: 25th International Conference on Human-Computer Interaction. Computer and Information Science (CCIS), Springer, Copenhagen, Denmark (2023)

    Google Scholar 

  26. Zinbarg, R.E., Revelle, W., Yovel, I., Li, W.: Cronbach’s \(\alpha \), revelle’s \(\beta \), and mcdonald’s \(\omega \) h: Their relations with each other and two alternative conceptualizations of reliability. psychometrika 70(1), 123–133 (2005)

    Google Scholar 

Download references

Acknowledgments

We thank the partnership between the University of Campinas and the CI &T company and its financial support. We also thank the undergraduate students who participated in the evaluations conducted in this study.

Author information

Authors and Affiliations

  1. Institute of Computing, University of Campinas, Campinas, SP, Brazil

    Paula Jeniffer dos Santos Viriato, Rafael Roque de Souza, Leandro Aparecido Villas & Julio Cesar dos Reis

  2. Nucleus of Informatics Applied to Education, University of Campinas, Campinas, SP, Brazil

    Julio Cesar dos Reis

Authors
  1. Paula Jeniffer dos Santos Viriato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rafael Roque de Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leandro Aparecido Villas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Julio Cesar dos Reis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paula Jeniffer dos Santos Viriato .

Editor information

Editors and Affiliations

  1. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

  2. Hosei University, Tokyo, Japan

    Ayako Hashizume

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

dos Santos Viriato, P.J., Roque de Souza, R., Villas, L.A., dos Reis, J.C. (2023). A Framework for Humanization Evaluation in Chatbots. In: Kurosu, M., Hashizume, A. (eds) Human-Computer Interaction. HCII 2023. Lecture Notes in Computer Science, vol 14013. Springer, Cham. https://doi.org/10.1007/978-3-031-35602-5_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35602-5_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35601-8

  • Online ISBN: 978-3-031-35602-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation