Seong Jae Lee
Abstract
We present a method for inferring the behavior styles of character controllers from a small set of examples. We show that a rich set of behavior variations can be captured by determining the appropriate reward function in the reinforcement learning framework, and show that the discovered reward function can be applied to different environments and scenarios. We also introduce a new algorithm to recover the unknown reward function that improves over the original apprenticeship learning algorithm. We show that the reward function representing a behavior style can be applied to a variety of different tasks, while still preserving the key features of the style present in the given examples. We describe an adaptive process where an author can, with just a few additional examples, refine the behavior so that it has better generalization properties.
Supplemental Material
Available for Download
Learning Behavior Styles using Inverse Reinforcement Learning
- Abbeel, P., and Ng, A. Y. 2004. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st International Conference on Machine Learning, ACM Press. Google Scholar
Digital Library
- Abbeel, P., Dolgov, D., Ng, A., and Thrun, S. 2008. Apprenticeship learning for motion planning, with application to parking lot navigation. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE.Google Scholar
- Barber, C. B., Dobkin, D. P., and Huhdanpaa, H. 1995. The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software 22, 469--483. Google ScholarDigital Library
- Beaudoin, P., Coros, S., van de Panne, M., and Poulin, P. 2008. Motion-motif graphs. In Proceedings of the 2008 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, 117--126. Google ScholarDigital Library
- Bellman, R. E. 1957. Dynamic Programming. Princeton University Press. Google ScholarDigital Library
- Brand, M., and Hertzmann, A. 2000. Style machines. In Proceedings of SIGGRAPH 2000, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM, 183--192. Google ScholarDigital Library
- Coates, A., Abbeel, P., and Ng, A. Y. 2009. Apprenticeship learning for helicopter control. Communications of the ACM 52, 7, 97--105. Google ScholarDigital Library
- Funge, J., Tu, X., and Terzopoulos, D. 1999. Cognitive modeling: knowledge, reasoning and planning for intelligent characters. In Proceedings of SIGGRAPH 99, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM, 29--38. Google ScholarDigital Library
- Grochow, K., Martin, S. L., Hertzmann, A., and Popović, Z. 2004. Style-based inverse kinematics. ACM Transactions on Graphics 23, 3, 522--531. Google ScholarDigital Library
- Heck, R., and Gleicher, M. 2007. Parametric motion graphs. In Proceedings of the 2007 symposium on Interactive 3D graphics and games, ACM, 129--136. Google ScholarDigital Library
- Hsu, E., Pulli, K., and Popović, J. 2005. Style translation for human motion. ACM Transactions on Graphics 24, 3, 1082--1089. Google ScholarDigital Library
- Kovar, L., Gleicher, M., and Pighin, F. 2002. Motion graphs. In Proceedings of SIGGRAPH 2002, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM, 473--482. Google ScholarDigital Library
- Lau, M., and Kuffner, J. J. 2006. Precomputed search trees: Planning for interactive goal-driven animation. In Proceedings of the 2006 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, Eurographics Association, 299--308. Google ScholarDigital Library
- Lee, J., and Lee, K. 2006. Precomputing avatar behavior from human motion data. Graphics Models 68, 2, 158--174. Google ScholarDigital Library
- Lee, K. H., Choi, M. G., Hong, Q., and Lee, J. 2007. Group behavior from video: a data-driven approach to crowd simulation. In Proceedings of the 2007 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, Eurographics Association, 109--118. Google ScholarDigital Library
- Lee, Y., Lee, S., and Popović, Z. 2009. Compact character controllers. ACM Transaction on Graphics 28, 5, 169:1--169:8. Google ScholarDigital Library
- Liu, K., Hertzmann, A., and Popović, Z. 2005. Learning physics-based motion style with nonlinear inverse optimization. ACM Transactions on Graphics 24, 3 (Aug.), 1071--1081. Google ScholarDigital Library
- Lo, W., and Zwicker, M. 2008. Real-time planning for parameterized human motion. In Proceedings of the 2008 Eurographics / ACM SIGGRAPH Symposium on Computer Animation, Eurographics Association, 29--38. Google ScholarDigital Library
- McCann, J., and Pollard, N. 2007. Responsive characters from motion fragments. ACM Transactions on Graphics 26, 3 (July), 6:1--6:7. Google ScholarDigital Library
- Ng, A. Y., and Russell, S. 2000. Algorithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, Morgan Kaufmann, 663--670. Google ScholarDigital Library
- Reitsma, P. S. A., and Pollard, N. S. 2004. Evaluating motion graphs for character navigation. In Proceedings of the 2004 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, Eurographics Association, 89--98. Google ScholarDigital Library
- Russell, S. 1998. Learning agents for uncertain environments (extended abstract). In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, ACM Press, 101--103. Google ScholarDigital Library
- Shin, H. J., and Oh, H. S. 2006. Fat graphs: constructing an interactive character with continuous controls. In Proceedings of the 2006 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, Eurographics Association, 291--298. Google ScholarDigital Library
- Sung, M., Gleicher, M., and Chenney, S. 2004. Scalable behaviors for crowd simulation. Computer Graphics Forum 23, 3, 519--528.Google ScholarCross Ref
- Syed, U., Bowling, M., and Schapire, R. E. 2008. Apprenticeship learning using linear programming. In Proceedings of the 25th international conference on Machine learning, ACM, 1032--1039. Google ScholarDigital Library
- Treuille, A., Lee, Y., and Popović, Z. 2007. Near-optimal character animation with continuous control. ACM Transactions on Graphics 26, 3 (July), 7:1--7:7. Google ScholarDigital Library
- Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. 2008. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd national conference on Artificial intelligence, AAAI Press, 1433--1438. Google ScholarDigital Library
Index Terms
- Learning behavior styles with inverse reinforcement learning
Recommendations
Learning behavior styles with inverse reinforcement learning
SIGGRAPH '10: ACM SIGGRAPH 2010 papersWe present a method for inferring the behavior styles of character controllers from a small set of examples. We show that a rich set of behavior variations can be captured by determining the appropriate reward function in the reinforcement learning ...
A survey of inverse reinforcement learning
AbstractLearning from demonstration, or imitation learning, is the process of learning to act in an environment from examples provided by a teacher. Inverse reinforcement learning (IRL) is a specific form of learning from demonstration that attempts to ...
Online Inverse Reinforcement Learning Under Occlusion
AAMAS '19: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent SystemsInverse reinforcement learning (IRL) is the problem of learning the preferences of an agent from observing its behavior on a task. While this problem is witnessing sustained attention, the related problem of online IRL - where the observations are ...
Comments