Abstract
Evaluation is one of the major open problems in Interactive Digital Storytelling (IDS) research. As narrative systems grow in their capacities, the community needs a set of well-designed evaluation methods and criteria that can bring insights on the systems as well as the stories they provide. In this short paper, we examine existing evaluation methods in the area of generative narrative system, and identify several important properties of stories and reading that have so far been overlooked in empirical studies. We present our preliminary work of developing a more interdisciplinary evaluation approach that takes into account both the system and cultural aspects of the computational narrative system Riu.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bringsjord, S., Ferrucci, D.A.: Artificial Intelligence and Literary Creativity: Inside the Mind of BRUTUS, a Storytelling Machine. Lawrence Erlbaum, Hillsdale (2000)
Gervás, P.: Computational approaches to storytelling and creativity. AI Magazine 30(3), 49–62 (2009)
Höök, K., Sengers, P., Andersson, G.: Sense and sensibility: evaluation and interactive art. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 241–248 (2003)
Jordanous, A.: Evaluating evaluation: Assessing progress in computational creativity research. In: Proceedings of the Second International Conference on Computational Creativity (ICCC 2011), pp. 102–107 (2011)
Lebowitz, M.: Story-telling as planning and learning. Poetics 14(6), 483–502 (1985)
Mateas, M.: Expressive ai: A hybrid art and science practice. Leonardo 34(2), 147–153 (2001)
Meehan, J.: Tale-spin. In: Inside Computer Understanding: Five Programs Plus Miniatures. Lawrence Erlbaum Associates, New Haven (1981)
Mehta, M., Dow, S., Mateas, M., MacIntyre, B.: Evaluating a conversation-centered interactive drama. In: Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 1–8 (2007)
Miall, D.S., Kuiken, D.: Foregrounding, defamiliarization, and affect response to literary stories. Poetics 22, 389–407 (1994)
Ontañón, S., Zhu, J.: On the role of domain knowledge in analogy-based story generation. In: Proceedings of the Twenty-Second International Joint Conferences on Artificial Intelligence (IJCAI 2011), pp. 1717–1722 (2011)
Ontañón, S., Zhu, J.: Story and text generation through computational analogy in the riu system. In: Proceedings of AI and Interactive Digital Entertainment Conference (AIIDE 2010), pp. 51–56. AAAI Press (2010)
Penny, S.: Experience and abstraction: the arts and the logic of machines. In: Proceedings of PerthDAC 2007: 7th Digital Arts and Culture Conference (2007)
Pérez y Pérez, R., Sharples, M.: Mexica: A computer model of a cognitive account of creative writing. Journal of Experimental & Theoretical Artificial Intelligence 13(2), 119–139 (2001)
Riedl, M.: Narrative Generation: Balancing Plot and Character. Ph.D. thesis, North Carolina State University (2004)
Schoenau-Fog, H.: Hooked! – Evaluating Engagement as Continuation Desire in Interactive Narratives. In: Si, M., Thue, D., André, E., Lester, J.C., Tanenbaum, J., Zammitto, V. (eds.) ICIDS 2011. LNCS, vol. 7069, pp. 219–230. Springer, Heidelberg (2011)
Sengers, P.: Anti-Boxology: Agent Design in Cultural Context. Ph.D. thesis, Carnegie Mellon University (1998)
Thue, D., Bulitko, V., Spetch, M., Romanuik, T.: A computational model of perceived agency in video games. In: Proceedings of the Seventh Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 91–96 (2011)
Zhu, J.: Towards a new evaluation approach in computational narrative systems. In: Proceedings of the Third International Conference on Computational Creativity (ICCC 2012), pp. 150–154 (2012)
Zhu, J., Harrell, D.F.: Navigating the Two Cultures: a Critical Approach to AI-based Literary Practice, pp. 222–246. World Scientific, Singapore (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhu, J. (2012). Designing an Interdisciplinary User Evaluation for the Riu Computational Narrative System. In: Oyarzun, D., Peinado, F., Young, R.M., Elizalde, A., Méndez, G. (eds) Interactive Storytelling. ICIDS 2012. Lecture Notes in Computer Science, vol 7648. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34851-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-34851-8_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34850-1
Online ISBN: 978-3-642-34851-8
eBook Packages: Computer ScienceComputer Science (R0)