Abstract
AI technology is rapidly evolving and has vast potential for educational applications. This paper focuses on the technical iterations that took place as our project team developed a semi-intelligent conversational agent (CA) that uses speech recognition to fire spoken prompts to promote caregiver-child interaction as they read books aloud together. Situating this work in a design-based research methodology, the technical iterations reported here are part of the iterative “build” phase. (Easterday et al., 2018; Hoadley & Campos, 2022). The CA app promotes conversations between caregivers and children by listening to the human dyads as they read, matching their spoken words to marker words that would pinpoint the page of the storybook the dyads are reading, and playing a prompt corresponding to the page. The dynamic system that supports the app involves multiple components: web accessible services, data processing services, and human outputs, and it has gone through a combined seven iterations in three prototypes. Though a very small part of the DBR cycle, the technical iterations presented here have the potential to inform others interested in incorporating text-to-speech and other AI technologies into educational applications. We close with considerations for future directions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Towson, J.A., Fettig, A., Fleury, V.P., Abarca, D.L.: Dialogic reading in early childhood settings: a summary of the evidence base. Top. Early Childhood Spec. Educ. 37(3), 132–146 (2017). https://doi.org/10.1177/0271121417724875
Arnold, D.S., Whitehurst, G.J.: Accelerating language development through picture book reading: a summary of dialogic reading and its effect. In: Dickinson, D.K. (ed.) Bridges to literacy: Children, families, and schools, pp. 103–128. Blackwell Publishing, Malden (1994)
Doyle, B.G., Bramwell, W.: Promoting emergent literacy and social-emotional learning through dialogic reading. Read. Teach. 59(6), 554–564 (2006). https://doi.org/10.1598/RT.59.6.5
Hirsh-Pasek, K., Golinkoff, R.M.: Put your data to use: entering the real world of children and families. Perspect. Psychol. Sci. 14(1), 37–42 (2019). https://doi.org/10.1177/1745691618815161
Bustamante, A.S., et al.: More than just a game: transforming social inter-action and STEM play with Parkopolis. Dev. Psychol. 56(6), 1041–1056 (2020). https://doi.org/10.1037/dev0000923
Leech, K.A., Rowe, M.L.: An intervention to increase conversational turns between parents and young children. J. Child Lang. 48(2), 399–412 (2021). https://doi.org/10.1017/S0305000920000252
Leech, K.A., Wei, R., Harring, J.R., Rowe, M.L.: A brief parent-focused intervention to improve pre-schoolers’ conversational skills and school readiness. Dev. Psychol. 54(1), 15–28 (2008). https://doi.org/10.1037/dev0000411
Leech, K.A., Haber, A.S., Jalkh, Y., Corriveau, K.H.: Embedding scientific explanations into storybooks impacts children’s scientific discourse and learning. Front. Psychol. 11, 1016 (2020). https://doi.org/10.3389/fpsyg.2020.01016
Reeves, T.C.: Socially responsible educational technology research. Educ. Technol. 40(6), 19–28 (2000)
Anderson, T., Shattuck, J.: Design-based research: a decade of progress in education research? Educ. Res. 41(1), 16–25 (2012). https://doi.org/10.3102/0013189X11428813
Barab, S., Squire, K.: Design-based research: putting a stake in the ground. J. Learn. Sci. 13(1), 1–14 (2004)
Linder, S.M., Ramey, M.D., Zambak, S.: Predictors of school readiness in literacy and mathematics: a selective review of the literature. Early Childhood Res. Pract. 15(1) (2013). https://eric.ed.gov/?id=EJ1016152
Pham, H.: PyAudio: cross-platform audio I/O with PortAudio. Accessed 01 Aug 2019. https://people.csail.mit.edu/hubert/pyaudio/
Drew, J.: mpg321. Accessed 01 Aug 2019. https://mpg321.sourceforge.net/
MediaDevices: getUserMedia() method - Web APIs|MDN. https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia
Thompson, M., Lin, G.C., Schoenfeld, I., Uz-Bilgin, C., Leech, K.: Taking advice from a virtual agent: usability of an artificially intelligent smart speaker app for parent and child storybook reading. In: Filipiak, D., Kalir, J.H. (eds.) Proceedings of the 2022 Connected Learning Summit, pp. 100–108. Carnegie Mellon University, ETC Press, Virtual (2022). https://doi.org/10.57862/tg5r-ck86
Lin, G.C., Schoenfeld, I., Thompson, M., Xia, Y., Uz-Bilgin, C., Leech, K.: “What color are the fish’s scales?” Exploring parents’ and children’s natural interactions with a child-friendly virtual agent during storybook reading. In: Interaction Design and Children, pp. 185–195. ACM, Braga (2022). https://doi.org/10.1145/3501712.3529734
Smith, C., et al.: Interaction strategies for an affective conversational agent. In: Allbeck, J., Badler, N., Bickmore, T., Pelachaud, C., Safonova, A. (eds.) IVA 2010. LNCS (LNAI), vol. 6356, pp. 301–314. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15892-6_31
Smith, C., et al.: Interaction strategies for an affective conversational agent. Presence 20(5), 395–411 (2011). https://doi.org/10.1162/PRES_a_00063
Boutte, G.S., Hopkins, R., Waklatsi, T.: Perspectives, voices, and worldviews in frequently read children’s books. Early Educ. Dev. 19(6), 941–962 (2008). https://doi.org/10.1080/10409280802206643
Acknowledgment
This work was funded by the Chan Zuckerberg Foundation as part of the Reach Every Reader project. We would like to thank all the playtesters in and out of The Education Arcade who gave valuation feedback throughout the development process. We would also like to give special shout-outs to Louisa Rosenheck, who led the project early on and established the design values and principles, Scot Osterweil, who designed and created the character that embodied the conversational agent, Melissa Callaghan, who, besides being an exceptional researcher and designer, also voiced the first iteration (human version) of the conversational agent, and Dr. Cigdem Uz-Bilgin, without whom the formal usability testing would likely have fallen apart. Finally, the work would not have been possible without Dr. Kathryn Leech, whose dialogic reading strategies are the basis of our early literacy app, and Dr. James Kim, who provided invaluable insights toward the prompts the conversational agent should ask.
Author information
Authors and Affiliations
Contributions
Brandon Hanks developed the literacy app and performed writing – original draft and conceptualization of this paper. Grace Lin performed writing – original draft and conceptualization of this paper. Ilana Schoenfeld and Vishesh Kumar performed writing – review and editing.
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hanks, B., Lin, G.C., Schoenfeld, I., Kumar, V. (2023). From Spoken Words to Prompt Triggers: Technical Iterations of a Semi-Intelligent Conversational Agent to Promote Early Literacy. In: Zaphiris, P., et al. HCI International 2023 – Late Breaking Papers. HCII 2023. Lecture Notes in Computer Science, vol 14060. Springer, Cham. https://doi.org/10.1007/978-3-031-48060-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-48060-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-48059-1
Online ISBN: 978-3-031-48060-7
eBook Packages: Computer ScienceComputer Science (R0)