Skip to main content
SpringerLink
Log in

The Show Must Go On: On the Use of Embodiment, Space and Gesture in Computational Storytelling

  • Published:
New Generation Computing Aims and scope Submit manuscript

Abstract

Stories are made to be told, yet the computational generation of stories has principally focused on stories as textual artifacts, rather than on the telling, or indeed the performance, of stories. The performative aspect of stories, in which a teller brings a tale to life, requires more than the written word. We humans use our bodies to enact a story, through the apt use of motion, space, timing and gesture. This work explores the physical enactment of computer-generated stories using multiple robots, which narrate the tale, and take on different roles of characters within it. They use pantomime to enhance the drama of narrative events, and use naturalistic gestures for more subtle communicative effects. They also use space as a mirror for abstract concerns such as affect and social relations. The paper outlines the Scéalability framework for turning story artifacts into performances, and presents empirical findings on the effectiveness of various embodied strategies. In particular, we show that audiences are sensitive to the coherent use of space in embodied story-telling, and appreciate the schematic use of spatial movements as much as more culturally specific pantomime gestures. For the presented study, we focus on one dimension of spatial movement involving two robot actors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://vimeo.com/364645778.

  2. translated from Goldstein [13].

  3. youtu.be/0ZQewRzo7gU.

  4. youtu.be/d4xNMFXTZLU.

  5. To see the video materials for each condition, please visit tiny.cc/s21fiz.

References

  1. Behrend, T.S., Sharek, D.J., Meade, A.W., Wiebe, E.N.: The viability of crowdsourcing for survey research. Behav. Res. Methods 43(3), 800 (2011)

    Article  Google Scholar 

  2. Candello, H., Pinhanez, C., Pichiliani, M.C., Guerra, M.A., de Bayser, M.G: Having an animated coffee with a group of chatbots from the 19th century. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pp. D206. ACM (2018)

  3. Cienki, A.: Image schemas and gesture. In: From Perception to Meaning: Image Schemas in Cognitive Linguistics, vol. 29, pp. 421–442 (2005)

  4. Cienki, A., Müller, C.: Metaphor, gesture, and thought. The Cambridge Handbook of Metaphor and Thought, pp. 483–501 (2008)

  5. Costa, S., Brunete, A., Bae, B.-C., Mavridis, N.: Emotional storytelling using virtual and robotic agents. Int. J. Hum. Robot. 15(03), 1850006 (2018)

    Article  Google Scholar 

  6. Cristani, M., Paggetti, G., Vinciarelli, A., Bazzani, L., Menegaz, G., Murino, V.: Towards computational proxemics: inferring social relations from interpersonal distances. In: 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, pp. 290–297. IEEE (2011)

  7. Csapo, A., Gilmartin, E., Grizou, J., Han, J., Meena, R., Anastasiou, D., Jokinen, K., Wilcock, G.: Multimodal conversational interaction with a humanoid robot. In: 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom), pp. 667–672. IEEE (2012)

  8. Dehn, N.: Story generation after tale-spin. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence—Volume 1, IJCAI’81, pp. 16–18. Morgan Kaufmann Publishers Inc., San Francisco (1981)

  9. Di Paolo, E.A., Thompson, E.: The enactive approach. The Routledge handbook of embodied cognition, pp. 68–78 (2014)

  10. Fischer-Lichte, E.: Ästhetik des Performativen. Suhrkamp Verlag, Frankfurt a. M. (2012)

    Google Scholar 

  11. Gibson, J.J.: The perception of the visual world (1950)

  12. Gifford, R., O’Connor, B.: Nonverbal intimacy: clarifying the role of seating distance and orientation. J. Nonverbal Behav. 10(4), 207–214 (1986)

    Article  Google Scholar 

  13. Goldstein, K.: über die abhängigkeit der bewegungen von optischen vorgängen. Eur. Neurol. 54(1), 141–153 (1923)

    Article  Google Scholar 

  14. Hall, E.T.: A system for the notation of proxemic behavior 1. Am. Anthropol. 65(5), 1003–1026 (1963)

    Article  Google Scholar 

  15. Hampe, B., Grady, J.E.: From Perception to Meaning: Image Schemas in Cognitive Linguistics, vol. 29. Walter de Gruyter, Berlin (2005)

    Book  Google Scholar 

  16. Hassenzahl, M., Burmester, M., Koller, F.: Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. In: Mensch & computer 2003, pp. 187–196. Springer (2003)

  17. Hayes-Roth, B.: A blackboard architecture for control. Artif. Intell. 26(3), 251–321 (1985)

    Article  Google Scholar 

  18. Heider, F., Simmel, M.: An experimental study of apparent behavior. Am. J. Psychol. 57(2), 243–259 (1944)

    Article  Google Scholar 

  19. Hüttenrauch, H., Eklundh, K.S., Green, A., Topp, E.A.: Investigating spatial relationships in human-robot interaction. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5052–5059. IEEE (2006)

  20. Jacob, M., Magerko, B.: Creative arcs in improvised human-computer embodied performances. In: Proceedings of the 13th International Conference on the Foundations of Digital Games, pp. 1–6 (2018)

  21. Johnson, M.: The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason. University of Chicago Press, Chicago (1987)

    Book  Google Scholar 

  22. Kelly, S.D., Barr, D.J., Church, R.B., Lynch, K.: Offering a hand to pragmatic understanding: the role of speech and gesture in comprehension and memory. J. Memory Lang. 40(4), 577–592 (1999)

    Article  Google Scholar 

  23. Klein, S., Aeschlimann, J.F., Balsiger, D.F., Converse, S.L., Foster, M., Lao, R., Oakley, J.D., Smith, J, et al.: Automatic novel writing: a status report. Technical report, University of Wisconsin-Madison Department of Computer Sciences (1973)

  24. Kleist, B.H.W.: Über das Marionettentheater. E. Lichtenstein (1920)

  25. Lakoff, G.: Women, Fire, and Dangerous Things. University of Chicago Press, Chicago (2008)

    Google Scholar 

  26. Le, Q.A., Hanoune, S., Pelachaud, C.: Design and implementation of an expressive gesture model for a humanoid robot. In: 2011 11th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 134–140. IEEE (2011)

  27. Le, Q.A., Pelachaud, C.: Evaluating an expressive gesture model for a humanoid robot: experimental results. In: Submitted to 8th ACM/IEEE International Conference on Human-Robot Interaction (2012)

  28. Lebowitz, M.: Story-telling as planning and learning. Poetics 14(6), 483–502 (1985)

    Article  Google Scholar 

  29. Li, B., Scheherazade, M.R.: Crowd-powered interactive narrative generation. In: Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)

  30. Lin, C.-Y., Tseng, C.-K., Teng, W.-C., Lee, W.-C., Kuo, C.-H., Gu, H.-Y., Chung, K.-L., Fahn, C.-S.: The realization of robot theater: humanoid robots and theatric performance. In: 2009 International Conference on Advanced Robotics, pp. 1–6. IEEE (2009)

  31. Lloyd, D.M.: The space between us: a neurophilosophical framework for the investigation of human interpersonal space. Neurosci. Biobehav. Rev. 33(3), 297–304 (2009)

    Article  Google Scholar 

  32. Lu, D.V., Smart, W.D.: Human-robot interactions as theatre. In: 2011 RO-MAN, pp. 473–478. IEEE (2011)

  33. Magerko, B., Laird, J.: Building an interactive drama architecture. In: First International Conference on Technologies for Interactive Digital Storytelling and Entertainment, pp. 226–237 (2003)

  34. Mandler, J.M., Cánovas, C.P.: On defining image schemas. Lang. Cogn. 6(4), 510–532 (2014)

    Article  Google Scholar 

  35. Maturana, H.R., Varela, F.J.: Autopoiesis and Cognition: The Realization of the Living, vol. 42. Springer Science & Business Media, Berlin (1991)

    Google Scholar 

  36. Mead, R., Matarić, M.J.: Autonomous human-robot proxemics: socially aware navigation based on interaction potential. Auton. Robots 41(5), 1189–1201 (2017)

    Article  Google Scholar 

  37. Meehan, J.R.: Tale-spin, an interactive program that writes stories. In: Proceedings of the 5th International Joint Conference on Artificial Intelligence—Volume 1, IJCAI’77, pp. 91–98. Morgan Kaufmann Publishers Inc., San Francisco (1977)

  38. Meena, R., Jokinen, K., Wilcock, G.: Integration of gestures and speech in human-robot interaction. In: 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom), pp. 673–678. IEEE (2012)

  39. Merleau-Ponty, M.: Phenomenology of Perception. Motilal Banarsidass Publisher, Delhi (1996)

    Google Scholar 

  40. Mittelberg, I.: Gestures as image schemas and force gestalts: a dynamic systems approach augmented with motion-capture data analyses. Cogn. Semiot. 11(1) (2018)

  41. Montfort, N.: Curveship’s automatic narrative style. In: Proceedings of the 6th International Conference on Foundations of Digital Games, pp. 211–218. ACM (2011)

  42. Montfort, N., y Pérez, R.P., Harrell, D.F., Campana, A.: Slant: a blackboard system to generate plot, figuration, and narrative discourse aspects of stories. In: ICCC, pp. 168–175 (2013)

  43. Newen, A., De Bruin, L., Gallagher, S.: The Oxford Handbook of 4E Cognition. Oxford University Press, Oxford (2018)

    Book  Google Scholar 

  44. Pelachaud, C., Gelin, R., Martin, J., Le, Q.A.: Expressive gestures displayed by a humanoid robot during a storytelling application. In: New Frontiers in Human-Robot Interaction (AISB), Leicester (2010)

  45. Pérez, RPy, Sharples, M.: Mexica: a computer model of a cognitive account of creative writing. J. Exp. Theor. Artif. Intell. 13(2), 119–139 (2001)

    Article  Google Scholar 

  46. Pope, V.C., Dawes, R., Schweiger, F., Sheikh, A.: The geometry of storytelling: theatrical use of space for 360-degree videos and virtual reality. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 4468–4478. ACM (2017)

  47. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8) (2019)

  48. Riedl, M.O., Bulitko, V.: Interactive narrative: an intelligent systems approach. Ai Mag. 34(1), 67–67 (2013)

    Article  Google Scholar 

  49. Riedl, M.O., Young, R.M.: An intent-driven planner for multi-agent story generation. In: Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 186–193. IEEE Computer Society (2004)

  50. Ritschel, H., Aslan, I., Sedlbauer, D., André, E.: Irony man: augmenting a social robot with the ability to use irony in multimodal communication with humans. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 86–94. International Foundation for Autonomous Agents and Multiagent Systems (2019)

  51. Román, I.G., y Pérez, R.P.: Social mexica: a computer model for social norms in narratives. In: Proceedings of the International Conference on Computational Creativity, pp. 192–200 (2014)

  52. Striepe, H., Donnermann, M., Lein, M., Lugrin, B.: Modeling and evaluating emotion, contextual head movement and voices for a social robot storyteller. Int. J. Soc. Robot. 1–17 (2019)

  53. Striepe, H., Lugrin, B.: There once was a robot storyteller: measuring the effects of emotion and non-verbal behaviour. In: International Conference on Social Robotics, pp. 126–136. Springer (2017)

  54. Trinh, H., Ring, L., Bickmore, T.: Dynamicduo: co-presenting with virtual agents. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1739–1748. ACM (2015)

  55. Turner, S.R.: Minstrel: A Computer Model of Creativity and Storytelling. PhD thesis, Los Angeles. UMI Order no. GAX93-19933 (1993)

  56. Veale, T.: Fighting words and antagonistic worlds. In: Proceedings of the Third Workshop on Metaphor in NLP, pp. 87–94 (2015)

  57. Veale, T.: Déja vu all over again. In: Proceedings of the International Conference on Computational Creativity (2017)

  58. Veale, T., Conway, A., Collins, B.: The challenges of cross-modal translation: english to sign language translation in the zardoz system. Mach. Transl. 13(1), 81–106 (1998)

    Article  Google Scholar 

  59. Veale, T., Keane, M.T.: Conceptual scaffolding: a spatially founded meaning representation for metaphor comprehension. Comput. Intell. 8(3), 494–519 (1992)

    Article  Google Scholar 

  60. Veale, T., Wicke, P., Mildner, T.: Duets ex machina: on the performative aspects of “double acts” in computational creativity. In: Proceedings of the International Conference on Computational Creativity (2019)

  61. Wicke, P., Veale, T.: Interview with the robot: question-guided collaboration in a storytelling system. In: ICCC, pp. 56–63 (2018)

  62. Wicke, P., Veale, T.: Storytelling by a show of hands: a framework for interactive embodied storytelling in robotic agents. In: Proc. of AISB’18, the Conf. on Artificial Intelligence and Simulated Behaviour, pp. 49–56 (2018)

  63. Wilcock, G.: Wikitalk: a spoken wikipedia-based open-domain knowledge access system. In: Proceedings of the Workshop on Question Answering for Complex Domains, pp. 57–70 (2012)

  64. Young, R.M.: Notes on the use of plan structures in the creation of interactive plot. In: AAAI Fall Symposium on Narrative Intelligence, pp. 164–167 (1999)

Download references

Author information

Authors and Affiliations

  1. University College Dublin, Dublin, Ireland

    Philipp Wicke & Tony Veale

Authors
  1. Philipp Wicke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tony Veale
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Wicke.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wicke, P., Veale, T. The Show Must Go On: On the Use of Embodiment, Space and Gesture in Computational Storytelling. New Gener. Comput. 38, 565–592 (2020). https://doi.org/10.1007/s00354-020-00106-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00354-020-00106-y

Keywords

Search

Navigation