Abstract
We present a computational framework for robotic animation of real-world string puppets. Also known as marionettes, these articulated figures are typically brought to life by human puppeteers. The puppeteer manipulates rigid handles that are attached to the puppet from above via strings. The motions of the marionette are therefore governed largely by gravity, the pull forces exerted by the strings, and the internal forces arising from mechanical articulation constraints. This seemingly simple setup conceals a very challenging and nuanced control problem, as marionettes are, in fact, complex coupled pendulum systems. Despite this, in the hands of a master puppeteer, marionette animation can be nothing short of mesmerizing. Our goal is to enable autonomous robots to animate marionettes with a level of skill that approaches that of human puppeteers. To this end, we devise a predictive control model that accounts for the dynamics of the marionette and kinematics of the robot puppeteer. The input to our system consists of a string puppet design and a target motion, and our trajectory planning algorithm computes robot control actions that lead to the marionette moving as desired. We validate our methodology through a series of experiments conducted on an array of marionette designs and target motions. These experiments are performed both in simulation and using a physical robot, the human-sized, dual arm ABB YuMi® IRB 14000.
Supplemental Material
- Thomas Auzinger, Wolfgang Heidrich, and Bernd Bickel. 2018. Computational design of nanostructural color for additive manufacturing. ACM Trans. Graph. 37, 4 (2018), 159:1--159:16. Google ScholarDigital Library
- Yunfei Bai, Wenhao Yu, and C. Karen Liu. 2016. Dexterous Manipulation of Cloth. Comput. Graph. Forum 35, 2 (2016), 523--532.Google ScholarCross Ref
- Jernej Barbič, Marco da Silva, and Jovan Popović. 2009. Deformable Object Animation Using Reduced Optimal Control. ACM Trans. Graph. 28, 3, Article 53 (July 2009), 9 pages. Google ScholarDigital Library
- James M. Bern, Kai-Hung Chang, and Stelian Coros. 2017. Interactive design of animated plushies. ACM Trans. Graph. 36, 4 (2017), 80:1--80:11. Google ScholarDigital Library
- M. Bernard and K. Kondak. 2009. Generic slung load transportation system using small size helicopters. In 2009 IEEE International Conference on Robotics and Automation. 3258--3264. Google ScholarDigital Library
- Morten Bisgaard, Jan Bendtsen, and Anders La Cour-Harbo. 2009. Modelling of Generic Slung Load System. In AIAA Modeling and Simulation Technologies Conference and Exhibit. American Institute of Aeronautics and Astronautics.Google Scholar
- Yang Cao, Shengtai Li, Linda Petzold, and Radu Serban. 2003. Adjoint sensitivity analysis for differential-algebraic equations: The adjoint DAE system and its numerical solution. SIAM Journal on Scientific Computing 24, 3 (2003), 1076--1089. Google ScholarDigital Library
- I.-Ming Chen, Raymond Tay, Shusong Xing, and Song Huat Yeo. 2004. Marionette: From Traditional Manipulation to Robotic Manipulation. In International Symposium on History of Machines and Mechanisms. Springer, Dordrecht, 119--133.Google ScholarCross Ref
- Alexander Clegg, Jie Tan, Greg Turk, and C. Karen Liu. 2015. Animating human dressing. ACM Trans. Graph. 34, 4 (2015), 116:1--116:9. Google ScholarDigital Library
- Alexander Clegg, Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. 2018. Learning to dress: synthesizing human dressing motion via deep reinforcement learning. ACM Trans. Graph. 37, 6 (2018), 179:1--179:10. Google ScholarDigital Library
- Michael F. Cohen. 1992. Interactive Spacetime Control for Animation. SIGGRAPH Comput. Graph. 26, 2 (July 1992), 293--302. Google ScholarDigital Library
- Stelian Coros, Bernhard Thomaszewski, Gioacchino Noris, Shinjiro Sueda, Moira Forberg, Robert W. Sumner, Wojciech Matusik, and Bernd Bickel. 2013. Computational design of mechanical characters. ACM Trans. Graph. 32, 4 (2013), 83:1--83:12. Google ScholarDigital Library
- Cédric De Crousaz, Farbod Farshidian, and Jonas Buchli. 2014. Aggressive optimal control for agile flight with a slung load. In in IROS 2014 Workshop on Machine Learning in Planning and Control of Robot Motion.Google Scholar
- Tao Du, Adriana Schulz, Bo Zhu, Bernd Bickel, and Wojciech Matusik. 2016. Computational multicopter design. ACM Trans. Graph. 35, 6 (2016), 227:1--227:10. http://dl.acm.org/citation.cfm?id=2982427 Google ScholarDigital Library
- Simon Duenser, James M. Bern, Roi Poranne, and Stelian Coros. 2018. Interactive Robotic Manipulation of Elastic Objects. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1--5, 2018. 3476--3481.Google Scholar
- A. Faust, I. Palunko, P. Cruz, R. Fierro, and L. Tapia. 2013. Learning swing-free trajectories for UAVs with a suspended load. In 2013 IEEE International Conference on Robotics and Automation. 4902--4909.Google Scholar
- Damien Gauge, Stelian Coros, Sandro Mani, and Bernhard Thomaszewski. 2014. Interactive Design of Modular Tensegrity Characters. In The Eurographics / ACM SIGGRAPH Symposium on Computer Animation, SCA 2014, Copenhagen, Denmark, 2014. 131--138. Google ScholarDigital Library
- Moritz Geilinger, Roi Poranne, Ruta Desai, Bernhard Thomaszewski, and Stelian Coros. 2018. Skaterbots: Optimization-based Design and Motion Synthesis for Robotic Creatures with Legs and Wheels. In Proceedings of ACM SIGGRAPH, ACM Transactions on Graphics (TOG) (Ed.), Vol. 37. ACM. Google ScholarDigital Library
- Gaël Guennebaud, Benoît Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org.Google Scholar
- John Hauser. 2002. A Projection Operator Approach To The Optimization Of Trajectory Functionals. IFAC Proceedings Volumes 35, 1 (2002), 377 -- 382. 15th IFAC World Congress.Google ScholarCross Ref
- Richard H. Jackson and Garth P. Mccormick. 1988. Second-order Sensitivity Analysis in Factorable Programming: Theory and Applications. Math. Program. 41, 1--3 (May 1988), 1--27. Google ScholarDigital Library
- E. Johnson and T. D. Murphey. 2007. Dynamic Modeling and Motion Planning for Marionettes: Rigid Bodies Articulated by Massless Strings. In Proceedings 2007 IEEE International Conference on Robotics and Automation. 330--335.Google Scholar
- Mickaël Ly, Romain Casati, Florence Bertails-Descoubes, Mélina Skouras, and Laurence Boissieux. 2018. Inverse Elastic Shell Design with Contact and Friction. In SIGGRAPH Asia 2018 Technical Papers (SIGGRAPH Asia '18). ACM, New York, NY, USA, Article 201, 16 pages. Google ScholarDigital Library
- Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus H. Gross. 2011. Example-based elastic materials. ACM Trans. Graph. 30, 4 (2011), 72:1--72:8. Google ScholarDigital Library
- Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid Control Using the Adjoint Method. In ACM SIGGRAPH 2004 Papers (SIGGRAPH '04). ACM, New York, NY, USA, 449--456. Google ScholarDigital Library
- Vittorio Megaro, Bernhard Thomaszewski, Maurizio Nitti, Otmar Hilliges, Markus H. Gross, and Stelian Coros. 2015. Interactive design of 3D-printable robotic creatures. ACM Trans. Graph. 34, 6 (2015), 216:1--216:9. Google ScholarDigital Library
- Vittorio Megaro, Jonas Zehnder, Moritz Bächer, Stelian Coros, Markus H. Gross, and Bernhard Thomaszewski. 2017. A computational design tool for compliant mechanisms. ACM Trans. Graph. 36, 4 (2017), 82:1--82:12. Google ScholarDigital Library
- E. Mizutani. 2015. On Pantoja's problem allegedly showing a distinction between differential dynamic programming and stagewise Newton methods. Internat. J. Control 88, 9 (2015), 1702--1711.Google ScholarCross Ref
- Todd D Murphey and E. Johnson. 2011. Control aesthetics in software architecture for robotic marionettes. In American Control Conference (ACC), 2011. IEEE, IEEE, 3825--3830.Google Scholar
- D. M. Murray and S. J. Yakowitz. 1984. Differential dynamic programming and Newton's method for discrete optimal control problems. Journal of Optimization Theory and Applications 43, 3 (01 Jul 1984), 395--414.Google ScholarCross Ref
- I. Palunko, A. Faust, P. Cruz, L. Tapia, and R. Fierro. 2013. A reinforcement learning approach towards autonomous suspended load manipulation using aerial robots. In 2013 IEEE International Conference on Robotics and Automation. 4896--4901.Google Scholar
- Zherong Pan and Dinesh Manocha. 2018. Active Animations of Reduced Deformable Models with Environment Interactions. ACM Trans. Graph. 37, 3, Article 36 (Aug. 2018), 17 pages. Google ScholarDigital Library
- J. F. A. De O. Pantoja. 1988. Differential dynamic programming and Newton's method. Internat. J. Control 47, 5 (1988), 1539--1553.Google ScholarCross Ref
- Jesús Pérez, Miguel A. Otaduy, and Bernhard Thomaszewski. 2017. Computational design and automated fabrication of kirchhoff-plateau surfaces. ACM Trans. Graph. 36, 4 (2017), 62:1--62:12. Google ScholarDigital Library
- J. Schultz and T. Murphey. 2012. Trajectory generation for underactuated control of a suspended mass. In 2012 IEEE International Conference on Robotics and Automation. 123--129.Google Scholar
- Adriana Schulz, Cynthia R. Sung, Andrew Spielberg, Wei Zhao, Robin Cheng, Eitan Grinspun, Daniela Rus, and Wojciech Matusik. 2017. Interactive robogami: An end-to-end system for design of robots with ground locomotion. I. J. Robotics Res. 36, 10 (2017), 1131--1147. Google ScholarDigital Library
- Christian Schulz, Christoph von Tycowicz, Hans-Peter Seidel, and Klaus Hildebrandt. 2014. Animating Deformable Objects Using Sparse Spacetime Constraints. ACM Trans. Graph. 33, 4, Article 109 (July 2014), 10 pages. Google ScholarDigital Library
- Mélina Skouras, Bernhard Thomaszewski, Stelian Coros, Bernd Bickel, and Markus H. Gross. 2013. Computational design of actuated deformable characters. ACM Trans. Graph. 32, 4 (2013), 82:1--82:10. Google ScholarDigital Library
- Peng Song, Xiaofei Wang, Xiao Tang, Chi-Wing Fu, Hongfei Xu, Ligang Liu, and Niloy J. Mitra. 2017. Computational design of wind-up toys. ACM Trans. Graph. 36, 6 (2017), 238:1--238:13. Google ScholarDigital Library
- Koushil Sreenath and Vijay Kumar. 2013. Dynamics, Control and Planning for Cooperative Manipulation of Payloads Suspended by Cables from Multiple Quadrotor Robots.Google Scholar
- K. Sreenath, N. Michael, and V. Kumar. 2013. Trajectory generation and control of a quadrotor with a cable-suspended load - A differentially-flat hybrid system. In 2013 IEEE International Conference on Robotics and Automation. 4888--4895.Google Scholar
- Bernhard Thomaszewski, Stelian Coros, Damien Gauge, Vittorio Megaro, Eitan Grinspun, and Markus H. Gross. 2014. Computational design of linkage-based characters. ACM Trans. Graph. 33, 4 (2014), 64:1--64:9. Google ScholarDigital Library
- Emanuel Todorov. 2014. Convex and analytically-invertible dynamics with contacts and constraints: Theory and implementation in MuJoCo. In 2014 IEEE International Conference on Robotics and Automation, ICRA 2014, Hong Kong, China, May 31 - June 7, 2014. 6054--6061.Google ScholarCross Ref
- Andrew Witkin and Michael Kass. 1988. Spacetime Constraints. SIGGRAPH Comput. Graph. 22, 4 (June 1988), 159--168. Google ScholarDigital Library
- Hongyi Xu, Espen Knoop, Stelian Coros, and Moritz Bächer. 2018. Bend-it: design and fabrication of kinetic wire characters. ACM Trans. Graph. 37, 6 (2018), 239:1--239:15. Google ScholarDigital Library
- Katsu Yamane, Jessica K. Hodgins, and H. Benjamin Brown. 2004. Controlling a motorized marionette with human motion capture data. International Journal of Humanoid Robotics 01, 04 (Dec. 2004), 651--669.Google ScholarCross Ref
- D. Zameroski, G. Starr, J. Wood, and R. Lumia. 2006. Swing-free trajectory generation for dual cooperative manipulators using dynamic programming. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. 1997--2003.Google Scholar
- Ran Zhang, Thomas Auzinger, Duygu Ceylan, Wilmot Li, and Bernd Bickel. 2017. Functionality-aware retargeting of mechanisms to 3D shapes. ACM Trans. Graph. 36, 4 (2017), 81:1--81:13. Google ScholarDigital Library
Index Terms
- PuppetMaster: robotic animation of marionettes
Recommendations
Mirror Puppeteering: Animating Toy Robots in Front of a Webcam
TEI '15: Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied InteractionMirror Puppeteering is a system for easily creating gestures ("animations") for robotic toys, custom robots, and virtual characters. Lay users can record animations by simply moving a robot's limbs in front of a webcam. Makers and hobbyists can use the ...
Sketch-n-Stretch: Sketching Animations Using Cutouts
Sketch-n-Stretch lets novices quickly and easily create 2D animations. The system has a seamless two-layer drawing interface that supports cutouts using a whiteboard metaphor. Cutout animations serve as building blocks to help users construct ...
Dynamic sprites: artistic authoring of interactive animations
Traditional methods for creating dynamic objects and characters from static drawings involve careful tweaking of animation curves and/or simulation parameters. Sprite sheets offer a more drawing-centric solution, but they do not encode timing ...
Comments