skip to main content
10.1145/3640794.3665565acmconferencesArticle/Chapter ViewAbstractPublication PagescuiConference Proceedingsconference-collections
Work in Progress

SUStory: Usability Evaluation of Conversational Interfaces for Children with a Narrative on a Game Board

Published: 08 July 2024 Publication History

Abstract

The increasing application of conversational agents as assistants and playmates for children brings about the need for evaluation methods tailored for children. The System Usability Scale (SUS) for kids is a well-established instrument for measuring subjective aspects of the usability for children. However, using the scale for children presents difficulties relating to children’s concentration and the biases observed when they respond to questionnaires. To make the questionnaire completion enjoyable for children and improve their engagement with the questionnaire, we rendered the adaptation of SUS for children in a game-board format. This paper motivates the presentation of the questionnaire in this way and discusses the use of this instrument in a case study with 35 children aged eight to twelve. We argue that this presentation helped engage with children and there was little evidence of extreme response bias. We conclude that this board format is a more appropriate way to present the SUS questionnaires to children in usability tests, which may also apply to different surveys including rating scales.

References

[1]
Tessa Aarts, Panos Markopoulos, Lars Giling, Tudor Vacaretu, and Sigrid Pillen. 2022. Snoozy: A chatbot-based sleep diary for children aged eight to twelve. In Proceedings of the 21st Annual ACM Interaction Design and Children Conference. 297–307.
[2]
Agnese Addone, Renato De Donato, Giuseppina Palmieri, Maria Angela Pellegrino, Andrea Petta, Vittorio Scarano, and Luigi Serra. 2021. Novelette, a usable visual storytelling digital learning environment. IEEE Access 9 (2021), 168850–168868.
[3]
Hans Baumgartner and Jan-Benedict EM Steenkamp. 2001. Response styles in marketing research: A cross-national investigation. Journal of marketing research 38, 2 (2001), 143–156.
[4]
Erin Beneteau, Olivia K Richards, Mingrui Zhang, Julie A Kientz, Jason Yip, and Alexis Hiniker. 2019. Communication breakdowns between families and Alexa. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–13.
[5]
Neelma Bhatti, Timothy L Stelter, Scott McCrickard, and Aisling Kelliher. 2021. Conversational user interfaces as assistive interlocutors for young children’s bilingual language acquisition. In Proceedings of the 2021 ACM International Conference on Interactive Media Experiences. 208–211.
[6]
Marc Brysbaert. 2019. How many words do we read per minute? A review and meta-analysis of reading rate. Journal of memory and language 109 (2019), 104047.
[7]
Fabio Catania, Micol Spitale, Giulia Cosentino, and Franca Garzotto. 2021. Conversational agents to promote children’s verbal communication skills. In Chatbot Research and Design: 4th International Workshop, CONVERSATIONS 2020, Virtual Event, November 23–24, 2020, Revised Selected Papers 4. Springer, 158–172.
[8]
Christine T Chambers and Charlotte Johnston. 2002. Developmental differences in children’s use of rating scales. Journal of pediatric psychology 27, 1 (2002), 27–36.
[9]
Shanshan Chen, Panos Markopoulos, and Jun Hu. 2023. Dozzz: Exploring Voice-based Sleep Experience Sampling for Children. In Proceedings of 17th EAI International Conference on Pervasive Computing Technologies for Healthcare.
[10]
Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards, 2019. The state of speech in HCI: Trends, themes and challenges. Interacting with computers 31, 4 (2019), 349–371.
[11]
Radhika Garg and Subhasree Sengupta. 2020. He is just like me: a study of the long-term use of smart speakers by parents and children. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1–24.
[12]
Guiseppe Getto. 2020. The story/test/story method: A combined approach to usability testing and contextual inquiry. Computers and Composition 55 (2020), 102548.
[13]
Guiseppe Getto and Christina Moore. 2017. Mapping personas: Designing UX relationships for an online coastal atlas. Computers and Composition 43 (2017), 15–34.
[14]
Jeffrey Goldstein. 2012. Play in children’s development, health and well-being. Toy Industries of Europe Brussels.
[15]
Thalia R Goldstein, Brittany N Thompson, and Pallavi Kanumuru. 2022. Do embodiment and fictionality affect young children’s learning?Journal of Experimental Child Psychology 213 (2022), 105275.
[16]
Michal Gordon and Cynthia Breazeal. 2015. Designing a virtual assistant for in-car child entertainment. In Proceedings of the 14th International Conference on Interaction Design and Children. 359–362.
[17]
Eric A Greenleaf. 1992. Measuring extreme response style. Public Opinion Quarterly 56, 3 (1992), 328–351.
[18]
Jean-Xavier Guinard. 2000. Sensory and consumer testing with children. Trends in Food Science & Technology 11, 8 (2000), 273–283.
[19]
Sandra Hale. 1990. A global developmental trend in cognitive processing speed. Child development 61, 3 (1990), 653–663.
[20]
Tina M Hickey and Anne-Marie de Mejia. 2014. Immersion education in the early years: A special issue., 131–143 pages.
[21]
Berk Laura. 2018. Exploring Child & Adolescent Development. Pearson.
[22]
Jingya Li, Tengjia Zuo, Erik Van Der Spek, and Jun Hu. 2024. Animated Scale: Adaption of the Motivational Scale for User Testing with Children. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 1–6.
[23]
Silvia B Lovato, Anne Marie Piper, and Ellen A Wartella. 2019. Hey Google, do unicorns exist? Conversational agents as a path to answers to children’s questions. In Proceedings of the 18th ACM international conference on interaction design and children. 301–313.
[24]
Karen Markey, Fritz Swanson, Andrea Jenkins, Brian J Jennings, Beth St. Jean, Victor Rosenberg, Xingxing Yao, and Robert L Frost. 2008. Designing and testing a web-based board game for teaching information literacy skills and concepts. Library Hi Tech 26, 4 (2008), 663–681.
[25]
Panos Markopoulos and Mathilde Bekker. 2003. On the assessment of usability testing methods for children. Interacting with computers 15, 2 (2003), 227–243.
[26]
Panos Markopoulos, Janet C Read, Stuart MacFarlane, and Johanna Hoysniemi. 2008. Evaluating children’s interactive products: principles and practices for interaction designers. Elsevier.
[27]
Ivonne Monarca, Franceli L Cibrian, Angel Mendoza, Gillian Hayes, and Monica Tentori. 2020. Why doesn’t the conversational agent understand me? a language analysis of children speech. In Adjunct proceedings of the 2020 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2020 ACM international symposium on wearable computers. 90–93.
[28]
Cheryl K Olson. 2010. Children’s motivations for video game play in the context of normal development. Review of general Psychology 14, 2 (2010), 180–187.
[29]
Jaehyun Park, Noha A Mostafa, and Hyo-Joo Han. 2020. “StoryWeb”: A storytelling-based knowledge-sharing application among multiple stakeholders. Creativity and Innovation Management 29, 2 (2020), 224–236.
[30]
Celia Pearce. 2006. Games as art: The aesthetics of play. Visible Language 40, 1 (2006), 66.
[31]
Cynthia Putnam, Melisa Puthenmadom, Marjorie Ann Cuerdo, Wanshu Wang, and Nathaniel Paul. 2020. Adaptation of the system usability scale for user testing with children. In Extended abstracts of the 2020 CHI conference on human factors in computing systems. 1–7.
[32]
Diana Pérez-Marín and Ismael Pascual-Nieto. 2013. An exploratory study on how children interact with pedagogic conversational agents. Behaviour & Information Technology 32, 9 (2013), 955–964.
[33]
Christian Roth and Hartmut Koenitz. 2016. Evaluating the user experience of interactive digital narrative. (2016), 31–36.
[34]
Heather Smith. 2006. Playing to learn: A qualitative analysis of bilingual pupil-pupil talk during board game play. Language and Education 20, 5 (2006), 415–437.
[35]
Jennifer Ureta, Celina Iris Brito, Jilyan Bianca Dy, Kyle-Althea Santos, Winfred Villaluna, and Ethel Ong. 2020. At home with Alexa: a tale of two conversational agents. In Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8–11, 2020, Proceedings 23. Springer, 495–503.
[36]
David L Vannette and Jon A Krosnick. 2014. Answering questions: a comparison of survey satisficing and mindlessness. The Wiley Blackwell handbook of mindfulness (2014), 312–327.
[37]
Diana Xu, Janet C Read, Emanuela Mazzone, and Martin Brown. 2007. Designing and testing a tangible interface prototype. In Proceedings of the 6th international conference on Interaction design and children. 25–28.
[38]
Ying Xu and Mark Warschauer. 2019. Young children’s reading and learning with conversational agents. In Extended abstracts of the 2019 CHI conference on human factors in computing systems. 1–8.
[39]
Bieke Zaman, Vero Vanden Abeele, and Dirk De Grooff. 2013. Measuring product liking in preschool children: An evaluation of the Smileyometer and This or That methods. International Journal of Child-Computer Interaction 1, 2 (2013), 61–70.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CUI '24: Proceedings of the 6th ACM Conference on Conversational User Interfaces
July 2024
616 pages
ISBN:9798400705113
DOI:10.1145/3640794
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2024

Check for updates

Author Tags

  1. SUS
  2. children
  3. conversational agents
  4. storytelling
  5. usability testing

Qualifiers

  • Work in progress
  • Research
  • Refereed limited

Conference

CUI '24
Sponsor:
CUI '24: ACM Conversational User Interfaces 2024
July 8 - 10, 2024
Luxembourg, Luxembourg

Acceptance Rates

Overall Acceptance Rate 34 of 100 submissions, 34%

Upcoming Conference

CUI '25
ACM Conversational User Interfaces 2025
July 7 - 9, 2025
Waterloo , ON , Canada

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 79
    Total Downloads
  • Downloads (Last 12 months)79
  • Downloads (Last 6 weeks)10
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media