Abstract
Recent years have seen growing interest in data-driven approaches to personalized interactive narrative generation and drama management. Reinforcement learning (RL) shows particular promise for training policies to dynamically shape interactive narratives based on corpora of player-interaction data. An important open question is how to design reinforcement learning-based drama managers in order to make effective use of player interaction data, which is often expensive to gather and sparse relative to the vast state and action spaces required by drama management. We investigate an offline optimization framework for training modular reinforcement learning-based drama managers in an educational interactive narrative, Crystal Island. We leverage importance sampling to evaluate drama manager policies derived from different decompositional representations of the interactive narrative. Empirical results show significant improvements in drama manager quality from adopting an optimized modular RL decomposition compared to competing representations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
There are 13 AESs in Crystal Island. In this work, we focus on four AESs, which were chosen because they were the most commonly occurring in our training corpus.
References
Rowe, J., Lester, J.: Improving student problem solving in narrative-centered learning environments: a modular reinforcement learning framework. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) 17th International Conference on Artificial Intelligence in Education. LNCS, vol. 9112, pp. 419–428. Springer, Heidelberg (2015)
Lee, S., Rowe, J., Mott, B., Lester, J.: A supervised learning framework for modeling director agent strategies in educational interactive narrative. IEEE Trans. Comput. Intell. AI Games 6, 203–215 (2014)
Li, B., Lee-Urban, S., Johnston, G., Riedl, M.: Story generation with crowdsourced plot graphs. In: 27th AAAI Conference on Artificial Intelligence, pp. 598–604 (2013)
Yu, H., Riedl, M.: Personalized interactive narratives via sequential recommendation of plot points. IEEE Trans. Comput. Intell. AI Games 6(2), 174–187 (2014)
Orkin, J., Roy, D.: Understanding speech in interactive narratives with crowd sourced data. In: 8th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 57–62 (2012)
Nelson, M., Roberts, D., Isbell, Jr., C., Mateas, M.: Reinforcement learning for declarative optimization-based drama management. In: 5th International Conference on Autonomous Agents and Multi-Agent Systems, pp. 775–782 (2006)
Roberts, D., Nelson, M., Isbell, C., Mateas, M., Littman, M.: Targeting specific distributions of trajectories in MDPs. In: 21st AAAI Conference on Artificial Intelligence, pp. 1213–1218 (2006)
Thue, D., Bulitko, V.: Procedural game adaptation: framing experience management as changing an MDP. In: Fifth Workshop on Intelligent Narrative Technologies, pp. 44–50 (2012)
Rowe, J., Mott, B., Lester, J.: Optimizing player experience in interactive narrative planning: a modular reinforcement learning approach. In: 10th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 160–166. AAAI Press, Menlo Park, CA (2014)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Peshkin, L., Shelton, C.: Learning from scarce experience. In: Proceedings of the Nineteenth International Conference on Machine Learning, pp. 498–505 (2002)
Riedl, M., Bulitko, V.: Interactive narrative: an intelligent systems approach. AI Mag. 34(1), 67–77 (2012)
McCoy, J. Treanor, M., Samuel, B., Mateas, M., Wardrip-Fruin, N.: Prom week: social physics as gameplay. In: 6th International Conference on Foundations of Digital Games, pp. 319–321 (2011)
Suttie, N., Louchart, S., Aylett, R., Lim, T.: Theoretical considerations towards authoring emergent narrative. In: Koenitz, H., Sezen, T.I., Ferri, G., Haahr, M., Sezen, D., Catak, G. (eds.) ICIDS 2013. LNCS, vol. 8230, pp. 205–216. Springer, Heidelberg (2013)
Porteous, J., Lindsay, A., Read, J., Truran, M., Cavazza. M.: Automated extension of narrative planning domains with antonymic operators. In: 14th International Conference on Autonomous Agents and Multiagent Systems, pp. 1547–1555 (2015)
Robertson, J., Young, R.: Automated gameplay generation from declarative world representations. In: 11th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 72–78 (2015)
Lamstein, A., Mateas, M.: Search-based drama management. In: 2004 AAAI Workshop on Challenges in Game Artificial Intelligence, pp. 103–107 (2004)
Singh, S.: Transfer of learning by composing solutions of elemental sequential tasks. Mach. Learn. 8, 323–339 (1992)
Dietterich, T.: Hierarchical reinforcement learning with the MAXQ value function decomposition. J. Artif. Intell. Res. 13, 227–303 (2000)
Chi, M., VanLehn, K., Litman, D., Jordan, P.: Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User Model. User-Adap. Inter. 21, 137–180 (2011)
Mandel, T., Liu, Y., Levine, S., Brunskill, E., Popovic, Z.: Offline policy evaluation across representations with applications to educational games. In: 13th International Conference on Autonomous Agents and Multi-Agent Systems, pp. 1077–1084 (2014)
Riezler, S., Maxwell, J.: On some pitfalls in automatic evaluation and significance testing for MT. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 57–64 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Wang, P., Rowe, J., Mott, B., Lester, J. (2016). Decomposing Drama Management in Educational Interactive Narrative: A Modular Reinforcement Learning Approach. In: Nack, F., Gordon, A. (eds) Interactive Storytelling. ICIDS 2016. Lecture Notes in Computer Science(), vol 10045. Springer, Cham. https://doi.org/10.1007/978-3-319-48279-8_24
Download citation
DOI: https://doi.org/10.1007/978-3-319-48279-8_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-48278-1
Online ISBN: 978-3-319-48279-8
eBook Packages: Computer ScienceComputer Science (R0)