Skip to main content
Springer Nature Link
Log in

Implementing and Evaluating Trustworthy Conversational Agents for Children

  • Conference paper
  • First Online:
Computer-Human Interaction Research and Applications (CHIRA 2024)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2370))

  • 55 Accesses

Abstract

Conversational Agents (CAs) have become increasingly popular in many settings, including households. However, despite the increasing frequency of children’s interactions with these systems, there is still little research on the ethical design of CAs, particularly for this special population. To address this gap, in this study we design, develop and evaluate a Child-Friendly CA for collaborative storytelling, implementing specific guidelines to ensure a trustworthy design for children based on key principles such as human agency, data privacy or transparency as outlined by the High-Level Expert Group on artificial intelligence (HLEG). To evaluate the trustworthiness of the Child-Friendly CA, designers and developers conduct a collaborative assessment by applying the Assessment List for Trustworthy Artificial Intelligence (ALTAI) using the Delphi methodology. Our results demonstrate that our Child-Friendly CA design improves the trustworthiness of the system and highlights the importance of designing CAs that consider the particularities of children’s interactions. Our findings contribute to the still scarce literature on trustworthy CAs and provide insights for developers striving to ensure a trustworthy experience for children.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://facctconference.org/.

  2. 2.

    https://european-union.europa.eu.

  3. 3.

    https://digital-strategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment.

  4. 4.

    https://assistant.google.com/intl/en-en/.

  5. 5.

    https://www.apple.com/siri/.

  6. 6.

    https://alexa.amazon.com.

  7. 7.

    https://openai.com/blog/introducing-chatgpt-and-whisper-apis.

  8. 8.

    https://github.com/guillaumekln/faster-whisper.

  9. 9.

    https://dialogflow.cloud.google.com.

  10. 10.

    https://cloud.google.com/dialogflow/es/docs/reference.

  11. 11.

    https://cloud.google.com/text-to-speech/docs/libraries?hl=es-419.

  12. 12.

    Bruno was chosen as a name that is similar in different languages such as Spanish, German or Italian.

  13. 13.

    The full dataset is available upon request.

References

  1. Australian AI Ethics Framework. Department of Industry, Science and Resources (2019). https://www.industry.gov.au/publications/australias-artificial-intelligence-ethics-framework

  2. Anderson, M., Anderson, S.L.: Machine ethics: creating an ethical intelligent agent. AI Mag. 28(4), 15–15 (2007)

    MATH  Google Scholar 

  3. Brey, T., Hanrieder, G., Heisterkamp, P., Hitzenberger, L., Regel-Brietzmann, P.: Issues in the evaluation of spoken dialogue systems-experience from the access project. In: LREC (2000). http://www.lrec-conf.org/proceedings/lrec2000/pdf/162.pdf

  4. Catania, F., et al.: Boris: a spoken conversational agent for music production for people with motor disabilities. In: CHItaly 2021: 14th Biannual Conference of the Italian SIGCHI Chapter, pp. 1–5 (2021)

    Google Scholar 

  5. Charisi, V., et al.: Artificial intelligence and the rights of the child: towards an integrated agenda for research and policy. Technical report, Joint Research Centre (Seville site) (2022)

    Google Scholar 

  6. Charisi, V., Dignum, V.: Operationalizing ai regulatory sandboxes for children’s rights and wellbeing. In: Régis, C., Denis, J.L., Axente, M.L., Kishimoto, A. (eds.) Human-Centered AI: A Multidisciplinary Perspective for Policy-Makers, Auditors, and Users, chap. 21. Routledge, New York (2024)

    Google Scholar 

  7. Charisi, V., Imai, T., Rinta, T., Nakhayenze, J.M., Gomez, R.: Exploring the concept of fairness in everyday, imaginary and robot scenarios: a cross-cultural study with children in japan and uganda. In: Proceedings of the 20th Annual ACM Interaction Design and Children Conference, pp. 532–536 (2021)

    Google Scholar 

  8. Del-Moral-Pérez, M.E., Villalustre-Martínez, L., Neira-Piñeiro, M.D.R.: Teachers’ perception about the contribution of collaborative creation of digital storytelling to the communicative and digital competence in primary education schoolchildren. Comput. Assisted Lang. Learn. 32(4), 342–365 (2019)

    Google Scholar 

  9. Deriu, J., et al.: Survey on evaluation methods for dialogue systems. Artif. Intell. Rev. 54, 755–810 (2021). https://doi.org/10.1007/s10462-020-09866-x

  10. Diederich, S., Brendel, A.B., Morana, S., Kolbe, L.: On the design of and interaction with conversational agents: an organizing and assessing review of human-computer interaction research. J. Assoc. Inf. Syst. 23(1), 96–138 (2022)

    MATH  Google Scholar 

  11. Dignum, V., Penagos, M., Pigmans, K., Vosloo, S.: Policy guidance on AI for children. Communications of UNICEF (2021)

    Google Scholar 

  12. Dybkjaer, L., Bernsen, N.O., Minker, W.: Evaluation and usability of multimodal spoken language dialogue systems. In: Speech Communication, vol. 43, pp. 33–54. Elsevier (2004). https://doi.org/10.1016/j.specom.2004.02.001

  13. Elgarf, M., Zojaji, S., Skantze, G., Peters, C.: Creativebot: a creative storyteller robot to stimulate creativity in children. In: Proceedings of the 2022 International Conference on Multimodal Interaction, pp. 540–548 (2022)

    Google Scholar 

  14. Engebak, I.M.H.: A digital game using collaborative storytelling to help children practice empathy. Master’s thesis, NTNU (2019)

    Google Scholar 

  15. Escobar-Planas, M., Charisi, V., Hupont, I., Martínez-Hinarejos, C.D., Gómez, E.: Towards children-centred trustworthy conversational agents. In: Chatbots-The AI-Driven Front-Line Services for Customers. IntechOpen (2023)

    Google Scholar 

  16. Escobar-Planas, M., Gómez, E., Martınez-Hinarejos, C.D.: Enhancing the design of a conversational agent for an ethical interaction with children. In: Proceedings of the IberSPEECH 2022, pp. 171–175 (2022). https://doi.org/10.21437/IberSPEECH.2022-35

  17. Escobar-Planas, M., Gómez, E., Martínez-Hinarejos, C.D.: Guidelines to develop trustworthy conversational agents for children. In: Proceedings of the ETHICOMP 2022, pp. 342–360 (2022)

    Google Scholar 

  18. Friedman, B., Nissenbaum, H.: Bias in computer systems. ACM Trans. Inf. Syst. (TOIS) 14(3), 330–347 (1996)

    Article  MATH  Google Scholar 

  19. Garg, R., Sengupta, S.: He is just like me: a study of the long-term use of smart speakers by parents and children. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4(1), 1–24 (2020)

    Article  MATH  Google Scholar 

  20. HLEG: Ethics guidelines for trustworthy ai. B-1049 Brussels (2019)

    Google Scholar 

  21. HLEG: The assessment list for trustworthy artificial intelligence (ALTAI). European Commission (2020)

    Google Scholar 

  22. Hopman, K., Richards, D., Norberg, M.N.: An embodied conversational agent to support wellbeing after injury: insights from a stakeholder inclusive design approach. In: International Conference on Persuasive Technology, pp. 161–175. Springer (2024)

    Google Scholar 

  23. İpek, Z.H., Gözüm, A.I.C., Papadakis, S., Kallogiannakis, M.: Educational applications of the chatgpt ai system: a systematic review research. Educ. Process: Int. J. 12(3), 26–55 (2023)

    MATH  Google Scholar 

  24. Lee, Y., Kim, T.S., Chang, M., Kim, J.: Interactive children’s story rewriting through parent-children interaction. In: Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pp. 62–71 (2022)

    Google Scholar 

  25. Likert, R.: A technique for the measurement of attitudes. Arch. Psychol. (1932)

    Google Scholar 

  26. Linstone, H.A., Turoff, M. (eds.): The Delphi Method. Addison-Wesley Reading, MA (1975)

    MATH  Google Scholar 

  27. Lovato, S.B., Piper, A.M., Wartella, E.A.: Hey google, do unicorns exist? Conversational agents as a path to answers to children’s questions. In: Proceedings of the IDC, pp. 301–313 (2019)

    Google Scholar 

  28. Lupetti, M.L., Hagens, E., Van Der Maden, W., Steegers-Theunissen, R., Rousian, M.: Trustworthy embodied conversational agents for healthcare: a design exploration of embodied conversational agents for the periconception period at erasmus mc. In: Proceedings of CUI, pp. 1–14 (2023)

    Google Scholar 

  29. Madiega, T.: Artificial intelligence act. European Parliament: European Parliamentary Research Service (2021)

    Google Scholar 

  30. Mehri, S., Eskenazi, M.: Unsupervised evaluation of interactive dialog with dialogpt. arXiv preprint arXiv:2006.12719, pp. 225–235 (2020). https://aclanthology.org/2020.sigdial-1.28/

  31. Ong, D.T., De Jesus, C.R., Gilig, L.K., Alburo, J.B., Ong, E.: A dialogue model for collaborative storytelling with children. In: ICCE 2018 - 26th International Conference on Computers in Education, Main Conference Proceedings, pp. 205–210 (2018)

    Google Scholar 

  32. Ong, E., Alburo, J.B., De Jesus, C.R., Gilig, L.K., Ong, D.T.: Challenges posed by voice interface to child-agent collaborative storytelling. In: 2019 22nd Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), pp. 1–6. IEEE (2019)

    Google Scholar 

  33. OpenAI: Chatgpt-4 (2024), large language model

    Google Scholar 

  34. Oswell, D.: The Agency of Children: From Family to Global Human Rights. Cambridge University Press, Cambridge (2013)

    Google Scholar 

  35. Packard, E.: The Cave of Time, ser. Choose Your Own Adventure. Bantam Books, New York (1979)

    Google Scholar 

  36. Radziwill, N.M., Benton, M.C.: Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579, pp. 25–36 (2017). https://arxiv.org/abs/1704.04579

  37. Rajamäki, J., Gioulekas, F., Rocha, P.A.L., Garcia, X.d.T., Ofem, P., Tyni, J.: ALTAI tool for assessing ai-based technologies: lessons learned and recommendations from shapes pilots. In: Healthcare, vol. 11, p. 1454. MDPI (2023)

    Google Scholar 

  38. Scharowski, N., Benk, M., Kühne, S.J., Wettstein, L., Brühlmann, F.: Certification labels for trustworthy AI: insights from an empirical mixed-method study. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 248–260 (2023)

    Google Scholar 

  39. Sciuto, A., Saini, A., Forlizzi, J., Hong, J.I.: Hey alexa, what’s up? A mixed-methods studies of in-home conversational agent usage. In: Proceedings of the 2018 Designing Interactive Systems Conference, pp. 857–868 (2018)

    Google Scholar 

  40. Slosiarová, N., Mesarčík, M., Jurkáček, P., Podroužek, J.: Trustworthy AI in dental care beyond artificial intelligence act (2023)

    Google Scholar 

  41. Stahl, B.C., Leach, T.: Assessing the ethical and social concerns of artificial intelligence in neuroinformatics research: an empirical test of the European union assessment list for trustworthy AI (ALTAI). AI Ethics 3(3), 745–767 (2023)

    Article  Google Scholar 

  42. Straten, C.L.v., Peter, J., Kühne, R., Barco, A.: Transparency about a robot’s lack of human psychological capacities: effects on child-robot perception and relationship formation. ACM Trans. Hum.-Robot Interact. (THRI) 9(2), 1–22 (2020)

    Google Scholar 

  43. Tolmeijer, S., Kneer, M., Sarasua, C., Christen, M., Bernstein, A.: Implementations in machine ethics: a survey. ACM Comput. Surv. (CSUR) 53(6), 1–38 (2020)

    Article  Google Scholar 

  44. UNESCO, C.: Recommendation on the ethics of artificial intelligence (2021)

    Google Scholar 

  45. UNICEF, Gomez, R., Charisi, V.: UNICEF Pilot study on Policy Guidance for AI and Child’s Rights. Office of Global Insight and Policy (2021). https://www.unicef.org/globalinsight/media/2206/file

  46. Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A.: Paradise: a framework for evaluating spoken dialogue agents. arXiv preprint cmp-lg/9704004, pp. 271–280 (1997). https://aclanthology.org/P97-1035/

  47. Wang, G., Zhao, J., Van Kleek, M., Shadbolt, N.: Informing age-appropriate ai: examining principles and practices of AI for children. In: Proceedings of the CHI, pp. 1–29 (2022)

    Google Scholar 

  48. Williams, R., Machado, C.V., Druga, S., Breazeal, C., Maes, P.: My doll says it’s ok a study of children’s conformity to a talking doll. In: Proceedings of IDC, pp. 625–631 (2018)

    Google Scholar 

  49. Xu, et al.: Mathkingdom: teaching children mathematical language through speaking at home via a voice-guided game. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2023)

    Google Scholar 

  50. Yeh, Y.T., Eskenazi, M., Mehri, S.: A comprehensive assessment of dialog evaluation metrics. arXiv preprint arXiv:2106.03706 pp. 15–33 (2021), https://aclanthology.org/2021.eancs-1.3/

  51. Zhang, L., Weitlauf, A.S., Amat, A.Z., Swanson, A., Warren, Z.E., Sarkar, N.: Assessing social communication and collaboration in autism spectrum disorder using intelligent collaborative virtual environments. J. Autism Dev. Disord. 50, 199–211 (2020)

    Article  Google Scholar 

  52. Zhang, Z., et al.: Storybuddy: a human-AI collaborative chatbot for parent-child interactive storytelling with flexible parental involvement. In: Proceedings of CHI, pp. 1–21 (2022)

    Google Scholar 

  53. Zicari, R.V., et al.: Co-design of a trustworthy ai system in healthcare: deep learning based skin lesion classifier. Front. Hum. Dyn. 3, 688152 (2021)

    Article  Google Scholar 

Download references

Acknowledgments

We thank all the stakeholders, developers and experts who supported the development and evaluation of the CAs, the stories creators and the translators who made multiple languages versions. During the preparation of this work, the authors used ChatGPT [33] to enhance language and readability. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the final version of the publication. This work was carried out with the support of the Joint Research Centre of the European Commission in the framework of the Collaborative Doctoral Partnership Agreement No. 35500.

Author information

Authors and Affiliations

  1. Universitat Politècnica de València, Camí de Vera, s/n, 46022, València, Spain

    Marina Escobar-Planas & Carlos-D. Martínez-Hinarejos

  2. Joint Research Centre, Calle Inca Garcilaso, 3, 41092, Seville, Spain

    Marina Escobar-Planas, Vicky Charisi & Emilia Gómez

  3. Universidad Pablo de Olavide, Ctra. de Utrera km 1, 41013, Seville, Spain

    Roberto Ruiz-Sánchez & Luis Merino

  4. Lambda Developments S.L., Calle grañon 12-4 1, 28050, Madrid, Spain

    Pedro Frau-Amar

Authors
  1. Marina Escobar-Planas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roberto Ruiz-Sánchez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pedro Frau-Amar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vicky Charisi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carlos-D. Martínez-Hinarejos
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Emilia Gómez
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Luis Merino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marina Escobar-Planas .

Editor information

Editors and Affiliations

  1. IT - Institute of Telecommunications, Instituto Superior Técnico, Lisbon, Portugal

    Hugo Plácido da Silva

  2. University of Turin, Turin, Italy

    Pietro Cipresso

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Escobar-Planas, M. et al. (2025). Implementing and Evaluating Trustworthy Conversational Agents for Children. In: Plácido da Silva, H., Cipresso, P. (eds) Computer-Human Interaction Research and Applications. CHIRA 2024. Communications in Computer and Information Science, vol 2370. Springer, Cham. https://doi.org/10.1007/978-3-031-82633-7_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-82633-7_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-82632-0

  • Online ISBN: 978-3-031-82633-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics