Skip to main content
SpringerLink
Log in

The Impact of Multi-character Story Distribution and Gesture on Children’s Engagement

  • Conference paper
  • First Online:
Interactive Storytelling (ICIDS 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11869))

Included in the following conference series:

  • 2939 Accesses

Abstract

Effective storytelling relies on engagement and interaction. This work develops an automated software platform for telling stories to children and investigates the impact of two design choices on children’s engagement and willingness to interact with the system: story distribution and the use of complex gesture. A storyteller condition compares stories told in a third person, narrator voice with those distributed between a narrator and first-person story characters. Basic gestures are used in all our storytellings, but, in a second factor, some are augmented with gestures that indicate conversational turn changes, references to other characters and prompt children to ask questions. An analysis of eye gaze indicates that children attend more to the story when a distributed storytelling model is used. Gesture prompts appear to encourage children to ask questions, something that children did, but at a relatively low rate. Interestingly, the children most frequently asked “why” questions. Gaze switching happened more quickly when the story characters began to speak than for narrator turns. These results have implications for future agent-based storytelling system research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aesop, Jones, V.S.V., Rackham, A., Chesterton, G.K.: Aesop’s Fables: A New Translation. W. Heinemann, Portsmouth (1933)

    Google Scholar 

  2. Aist, G., Kort, B., Reilly, R., Mostow, J., Picard, R.: Experimentally augmenting an intelligent tutoring system with human-supplied capabilities: adding human-provided emotional scaffolding to an automated reading tutor that listens. In: Proceedings of Fourth IEEE International Conference on Multimodal Interfaces, pp. 483–490. IEEE (2002)

    Google Scholar 

  3. Alemdag, E., Cagiltay, K.: A systematic review of eye tracking research on multimedia learning. Comput. Educ. 125, 413–428 (2018)

    Google Scholar 

  4. Alexander, K.L., Entwisle, D.R., Dauber, S.L.: First-grade classroom behavior: its short-and long-term consequences for school performance. Child Dev. 64(3), 801–814 (1993)

    Google Scholar 

  5. Andersen, J.F.: The relationship between teacher immediacy and teaching effectiveness. Ph.D. thesis, ProQuest Information & Learning (1979)

    Google Scholar 

  6. Andersen, J.F.: Instructor nonverbal communication: listening to our silent messages. New Direct. Teach. Learning 1986(26), 41–49 (1986)

    Google Scholar 

  7. Andersen, J.F., Norton, R.W., Nussbaum, J.F.: Three investigations exploring relationships between perceived teacher communication behaviors and student learning. Commun. Educ. 30(4), 377–392 (1981)

    Google Scholar 

  8. August, D., Hakuta, K.: Improving schooling for language minority students: a research agenda. A Research Agenda, Improving Schooling for Language Minority Students (1997)

    Google Scholar 

  9. Baylor, A.L.: Promoting motivation with virtual agents and avatars: role of visual presence and appearance. Philos. Trans. Roy. Soc. London B: Biol. Sci. 364(1535), 3559–3565 (2009)

    Google Scholar 

  10. Baylor, A.L., Kim, Y.: Pedagogical agent design: the impact of agent realism, gender, ethnicity, and instructional role. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 592–603. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30139-4_56

    Chapter  Google Scholar 

  11. Baylor, A.L., Ryu, J.: The effects of image and animation in enhancing pedagogical agent persona. J. Educ. Comput. Res. 28(4), 373–394 (2003)

    Google Scholar 

  12. Bloom, C.P., Fletcher, C.R., Broek, P.V.D., Reitz, L., Shapiro, B.P.: An on-line assessment of causal reasoning during comprehension. J. Mem. Cogn. 18, 65–71 (1990)

    Google Scholar 

  13. Breckinridge Church, R., Garber, P., Rogalski, K.: The role of gesture in memory and social communication. Gesture 7(2), 137–158 (2007)

    Google Scholar 

  14. Brooks-Gunn, J., Duncan, G.J.: The effects of poverty on children. Future Child. 7, 55–71 (1997)

    Google Scholar 

  15. Cassell, J.: Towards a model of technology and literacy development: story listening systems. J. Appl. Dev. Psychol. 25(1), 75–105 (2004)

    Google Scholar 

  16. Catts, H.W., Adlof, S.M., Hogan, T.P., Weismer, S.E.: Are specific language impairment and dyslexia distinct disorders? J. Speech Lang. Hear. Res. 48(6), 1378–1396 (2005)

    Google Scholar 

  17. Chang, A., Breazeal, C.: TinkRBook: shared reading interfaces for storytelling. In: Proceedings of the 10th International Conference on Interaction Design and Children, pp. 145–148. ACM (2011)

    Google Scholar 

  18. Chesebro, J.L.: Effects of teacher clarity and nonverbal immediacy on student learning, receiver apprehension, and affect. Commun. Educ. 52(2), 135–147 (2003)

    Google Scholar 

  19. Cunningham, A.E., Stanovich, K.E.: Early reading acquisition and its relation to reading experience and ability 10 years later. Dev. Psychol. 33(6), 934 (1997)

    Google Scholar 

  20. Duncan, G.J., et al.: School readiness and later achievement. Dev. Psychol. 43(6), 1428 (2007)

    Google Scholar 

  21. Elson, D.: DramaBank: annotating agency in narrative discourse. In: LREC, pp. 2813–2819 (2012)

    Google Scholar 

  22. Fletcher, C.R., Hummel, J.E., Marsolek, C.J.: Causality and the allocation of attention during comprehension. J. Exp. Psychol. 16(2), 233–240 (1990)

    Google Scholar 

  23. Goldin-Meadow, S., Singer, M.A.: From children’s hands to adults’ ears: gesture’s role in the learning process. Dev. Psychol. 39(3), 509 (2003)

    Google Scholar 

  24. Graesser, A.C., Singer, M., Trabasso, T.: Constructing inferences during narrative text comprehension. Psychol. Rev. 101(3), 371 (1994)

    Google Scholar 

  25. Hart, B., Risley, T.R.: Meaningful Differences in the Everyday Experience of Young American Children. Paul H Brookes Publishing, Baltimore (1995)

    Google Scholar 

  26. Hoff, E.: Environmental supports for language acquisition. Handb. Early Lit. Res. 2, 163–172 (2006)

    Google Scholar 

  27. Hoff, E., Naigles, L.: How children use input to acquire a lexicon. Child Dev. 73(2), 418–433 (2002)

    Google Scholar 

  28. Hoover, W.A., Gough, P.B.: The simple view of reading. Read. Writ. 2(2), 127–160 (1990)

    Google Scholar 

  29. Hyönä, J.: The use of eye movements in the study of multimedia learning. Learn. Instr. 20(2), 172–176 (2010)

    Google Scholar 

  30. Juel, C.: Learning to read and write: a longitudinal study of 54 children from first through fourth grades. J. Educ. Psychol. 80(4), 437 (1988)

    Google Scholar 

  31. Juel, C., Griffith, P.L., Gough, P.B.: Acquisition of literacy: a longitudinal study of children in first and second grade. J. Educ. Psychol. 78(4), 243 (1986)

    Google Scholar 

  32. Kanda, T., Hirano, T., Eaton, D., Ishiguro, H.: Interactive robots as social partners and peer tutors for children: a field trial. Hum.-Comput. interact. 19(1), 61–84 (2004)

    Google Scholar 

  33. Kennedy, J., Baxter, P., Belpaeme, T.: Nonverbal immediacy as a characterisation of social behaviour for human-robot interaction. Int. J. Social Robot. 9(1), 109–128 (2017)

    Google Scholar 

  34. Kim, Y., Baylor, A.L.: A social-cognitive framework for pedagogical agents as learning companions. Educ. Tech. Research Dev. 54(6), 569–596 (2006)

    Google Scholar 

  35. Korenman, S., Miller, J.E., Sjaastad, J.E.: Long-term poverty and child development in the united states: Results from the nlsy. Child Youth Serv. Rev. 17(1–2), 127–155 (1995)

    Google Scholar 

  36. Kory, J., Breazeal, C.: Storytelling with robots: learning companions for preschool children’s language development. In: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, RO-MAN 2014, pp. 643–648. IEEE (2014)

    Google Scholar 

  37. Lee, K.M., Peng, W., Jin, S.A., Yan, C.: Can robots manifest personality?: An empirical test of personality recognition, social responses, and social presence in human-robot interaction. J. Commun. 56(4), 754–772 (2006)

    Google Scholar 

  38. Literacy, D.E.: Report of the national early literacy panel Washington. DC National Institute for Literacy (2008)

    Google Scholar 

  39. Lukin, S.M., Reed, L.I., Walker, M.: Generating sentence planning variations for story telling. In: 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, p. 188 (2015)

    Google Scholar 

  40. Lukin, S.M., Walker, M.A.: A narrative sentence planner and structurer for domain independent, parameterizable storytelling. Dialogue Discourse 10(1), 34–86 (2019)

    Google Scholar 

  41. Mairesse, F., Walker, M.A.: Towards personality-based user adaptation: psychologically informed stylistic language generation. User Model. User-Adapt. Interact. 20(3), 227–278 (2010)

    Google Scholar 

  42. Mayer, R.E.: Cognitive theory of multimedia learning. Cambridge Handb. Multimed. Learn. 41, 31–48 (2005)

    Google Scholar 

  43. Mayer, R.E.: Using multimedia for e-learning. J. Comput. Assist. Learn. 33(5), 403–423 (2017)

    Google Scholar 

  44. McLoyd, V.C.: Socioeconomic disadvantage and child development. Am. Psychol. 53(2), 185 (1998)

    Google Scholar 

  45. McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)

    Google Scholar 

  46. Molina, A.I., Navarro, S., Ortega, M., Lacruz, M.: Evaluating multimedia learning materials in primary education using eye tracking. Comput. Stand. Interfaces 59(C), 45–60 (2018). https://doi.org/10.1016/j.csi.2018.02.004

    Article  Google Scholar 

  47. Mostow, J., et al.: Evaluation of an automated reading tutor that listens: Comparison to human tutoring and classroom instruction. J. Educ. Comput. Res. 29(1), 61–117 (2003)

    Google Scholar 

  48. NICHD Early Child Care Research Network, et al.: Child Care And Child Development: Results from the NICHD Study of Early Child Care and Youth Development. Guilford Press (2005)

    Google Scholar 

  49. Park, H.W., Gelsomini, M., Lee, J.J., Breazeal, C.: Telling stories to robots: the effect of backchanneling on a child’s storytelling. In: Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, pp. 100–108. ACM (2017)

    Google Scholar 

  50. Piwek, P., Hernault, H., Prendinger, H., Ishizuka, M.: T2D: generating dialogues between virtual agents automatically from text. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 161–174. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74997-4_16

    Chapter  Google Scholar 

  51. Piwek, P., Stoyanchev, S.: Generating expository dialogue from monologue: motivation, corpus and preliminary rules. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 333–336. Association for Computational Linguistics (2010)

    Google Scholar 

  52. Ritchie, S.J., Bates, T.C.: Enduring links from childhood mathematics and reading achievement to adult socioeconomic status. Psychol. Sci. 24(7), 1301–1308 (2013)

    Google Scholar 

  53. Rodrigues, P., Rosa, P.J.: Eye-tracking as a research methodology in educational context: a spanning framework. In: Eye-Tracking Technology Applications in Educational Research, pp. 1–26. IGI Global (2017)

    Google Scholar 

  54. Rodríguez, J.I., Plax, T.G., Kearney, P.: Clarifying the relationship between teacher nonverbal immediacy and student cognitive learning: affective learning as the central causal mediator. Commun. Educ. 45(4), 293–305 (1996)

    Google Scholar 

  55. Ryokai, K., Vaucelle, C., Cassell, J.: Virtual peers as partners in storytelling and literacy learning. J. Comput. Assist. Learn. 19(2), 195–208 (2003)

    Google Scholar 

  56. Snow, C.E., Burns, M.S., Griffin, P.: Preventing reading difficulties in young children committee on the prevention of reading difficulties in young children. National Research Council, Washington, DC (1998)

    Google Scholar 

  57. Sperry, D.E., Sperry, L.L., Miller, P.J.: Reexamining the verbal environments of children from different socioeconomic backgrounds. Child Dev. 90(4), 1303–1318 (2019). https://doi.org/10.1111/cdev.13072. https://onlinelibrary.wiley.com/doi/abs/10.1111/cdev.13072

    Article  Google Scholar 

  58. Staudte, M., Crocker, M.W., Heloir, A., Kipp, M.: The influence of speaker gaze on listener comprehension: contrasting visual versus intentional accounts. Cognition 133(1), 317–328 (2014)

    Google Scholar 

  59. Storch, S.A., Whitehurst, G.J.: Oral language and code-related precursors to reading: evidence from a longitudinal structural model. Dev. Psychol. 38(6), 934 (2002)

    Google Scholar 

  60. Takacs, Z.K., Bus, A.G.: Benefits of motion in animated storybooks for children visual attention and story comprehension. An eye-tracking study. Front. Psychol. 7, 1591 (2016)

    Google Scholar 

  61. Thompson, L.A., Massaro, D.W.: Children’s integration of speech and pointing gestures in comprehension. J. Exp. Child Psychol. 57(3), 327–354 (1994)

    Google Scholar 

  62. Torgesen, J.K.: Avoiding the devastating downward spiral: the evidence that early intervention prevents reading failure. Am. Educat. 28(3), 6–19 (2004)

    Google Scholar 

  63. Vellutino, F.R., Scanlon, D.M., Zhang, H.: Identifying reading disability based on response to intervention: evidence from early intervention research. In: Jimerson, S.R., Burns, M.K., VanDerHeyden, A.M. (eds.) Handbook of Response to Intervention, pp. 185–211. Springer, Boston (2007). https://doi.org/10.1007/978-0-387-49053-3_14

    Chapter  Google Scholar 

  64. Wertheimer, R.F., Moore, K.A., Hair, E.C., Croan, T.: Attending kindergarten and already behind: a statistical portrait of vulnerable young children. Child Trends Washington, DC (2003)

    Google Scholar 

  65. Westlund, J.K., Breazeal, C.: The interplay of robot language level with children’s language learning during storytelling. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts, pp. 65–66. ACM (2015)

    Google Scholar 

  66. Whittaker, S.: Theories and methods in mediated communication: Steve Whittaker. In: Handbook of Discourse Processes, pp. 246–289. Routledge (2003)

    Google Scholar 

  67. Zill, N.: Promoting educational equity and excellence in kindergarten. Transit. Kindergart. 67–105 (1999)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of California, Davis, Davis, CA, 95616, USA

    Harrison Jesse Smith, Brian K. Riley & Michael Neff

  2. University of California, Santa Cruz, Santa Cruz, CA, 95064, USA

    Lena Reed, Vrindavan Harrison & Marilyn Walker

Authors
  1. Harrison Jesse Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian K. Riley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lena Reed
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vrindavan Harrison
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marilyn Walker
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael Neff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Harrison Jesse Smith .

Editor information

Editors and Affiliations

  1. University of Utah, Salt Lake City, UT, USA

    Rogelio E. Cardona-Rivera

  2. Georgia Institute of Technology, Atlanta, GA, USA

    Anne Sullivan

  3. University of Utah, Salt Lake City, UT, USA

    R. Michael Young

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Smith, H.J., Riley, B.K., Reed, L., Harrison, V., Walker, M., Neff, M. (2019). The Impact of Multi-character Story Distribution and Gesture on Children’s Engagement. In: Cardona-Rivera, R., Sullivan, A., Young, R. (eds) Interactive Storytelling. ICIDS 2019. Lecture Notes in Computer Science(), vol 11869. Springer, Cham. https://doi.org/10.1007/978-3-030-33894-7_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-33894-7_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-33893-0

  • Online ISBN: 978-3-030-33894-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation