Skeleton-Driven Inbetweening of Bitmap Character Drawings
Article No.: 246, Pages 1 - 19
Abstract
One of the primary reasons for the high cost of traditional animation is the inbetweening process, where artists manually draw each intermediate frame necessary for smooth motion. Making this process more efficient has been at the core of computer graphics research for years, yet the industry has adopted very few solutions. Most existing solutions either require vector input or resort to tight inbetweening; often, they attempt to fully automate the process. In industry, however, keyframes are often spaced far apart, drawn in raster format, and contain occlusions. Moreover, inbetweening is fundamentally an artistic process, so the artist should maintain high-level control over it.
We address these issues by proposing a novel inbetweening system for bitmap character drawings, supporting both tight and far inbetweening. In our setup, the artist can control motion by animating a skeleton between the keyframe poses. Our system then performs skeleton-based deformation of the bitmap drawings into the same pose and employs discrete optimization and deep learning to blend the deformed images. Besides the skeleton and the two drawn bitmap keyframes, we require very little annotation.
However, deforming drawings with occlusions is complex, as it requires a piecewise smooth deformation field. To address this, we observe that this deformation field is smooth when the drawing is lifted into 3D. Our system therefore optimizes topology of a 2.5D partially layered template that we use to lift the drawing into 3D and get the final piecewise-smooth deformaton, effectively resolving occlusions.
We validate our system through a series of animations, qualitative and quantitative comparisons, and user studies, demonstrating that our approach consistently outperforms the state of the art and our results are consistent with the viewers' perception.
Code and data for our paper are available at http://www-labs.iro.umontreal.ca/~bmpix/inbetweening/.
References
[1]
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
[2]
Yunfei Bai, Danny M. Kaufman, C. Karen Liu, and Jovan Popović. 2016. Artist-directed dynamics for 2D animation. ACM Trans. Graph. 35, 4, Article 145 (jul 2016), 10 pages.
[3]
Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. 2019. Depth-Aware Video Frame Interpolation. In IEEE Conference on Computer Vision and Pattern Recognition.
[4]
N. Ben-Zvi, J. Bento, M. Mahler, J. Hodgins, and A. Shamir. 2016. Line-Drawing Video Stylization. Computer Graphics Forum 35, 6 (2016), 18--32. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.12729
[5]
Mikhail Bessmeltsev, Will Chang, Nicholas Vining, Alla Sheffer, and Karan Singh. 2015. Modeling Character Canvases from Cartoon Drawings. Transactions on Graphics (2015) 34, 5 (2015).
[6]
Mikhail Bessmeltsev and Justin Solomon. 2019. Vectorization of Line Drawings via Polyvector Fields. ACM Trans. Graph. 38, 1, Article 9 (Jan. 2019), 12 pages.
[7]
Kirill Brodt and Mikhail Bessmeltsev. 2022. Sketch2Pose: Estimating a 3D Character Pose from a Bitmap Sketch. ACM Transactions on Graphics 41, 4 (7 2022).
[8]
N. Burtnyk and M. Wein. 1976. Interactive Skeleton Techniques for Enhancing Motion Dynamics in Key Frame Animation. Commun. ACM 19, 10 (oct 1976), 564--569.
[9]
Leonardo Carvalho, Ricardo Marroquim, and Emilio Vital Brazil. 2017. DiLight: Digital light table - Inbetweening for 2D animations using guidelines. Computers & Graphics 65 (2017), 31--44.
[10]
Edwin Catmull. 1978. The Problems of Computer-Assisted Animation. SIGGRAPH Comput. Graph. 12, 3 (aug 1978), 348--353.
[11]
Jiawen Chen, Sylvain Paris, Jue Wang, Wojciech Matusik, Michael Cohen, and Frédo Durand. 2011. The video mesh: A data structure for image-based three-dimensional video editing. In 2011 IEEE International Conference on Computational Photography (ICCP). 1--8.
[12]
J. Chen, X. Zhu, M. Even, J. Basset, P. Bénard, and P. Barla. 2023. Efficient Interpolation of Rough Line Drawings. Computer Graphics Forum 42, 7 (2023), e14946. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14946
[13]
Shuhong Chen and Matthias Zwicker. 2022. Improving the Perceptual Quality of 2D Animation Interpolation. In Proceedings of the European Conference on Computer Vision.
[14]
Boris Dalstein, Rémi Ronfard, and Michiel van de Panne. 2015. Vector Graphics Animation with Time-Varying Topology. ACM Trans. Graph. 34, 4 (July 2015).
[15]
Marek Dvorožňák, Wilmot Li, Vladimir G. Kim, and Daniel Sýkora. 2018. ToonSynth: Example-Based Synthesis of Hand-Colored Cartoon Animations. ACM Transactions on Graphics 37, 4, Article 167 (2018).
[16]
Marek Dvorožňák, Daniel Sýkora, Cassidy Curtis, Brian Curless, Olga Sorkine-Hornung, and David Salesin. 2020. Monster Mash: A Single-View Approach to Casual 3D Modeling and Animation. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2020) 39, 6 (2020), 214.
[17]
Melvin Even, Pierre Bénard, and Pascal Barla. 2023. Non-linear Rough 2D Animation using Transient Embeddings. Computer Graphics Forum 42, 2 (2023), 411--425. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14771
[18]
Xinyi Fan, Amit H Bermano, Vladimir G Kim, Jovan Popović, and Szymon Rusinkiewicz. 2018. Tooncap: A layered deformable model for capturing poses from cartoon characters. In Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering. 1--12.
[19]
Tsukasa Fukusato and Akinobu Maejima. 2022. View-Dependent Deformation for 2.5-D Cartoon Models. IEEE Computer Graphics and Applications 42, 5 (2022), 66--75. Publisher Copyright: © 1981--2012 IEEE.
[20]
Olga Guţan, Shreya Hegde, Erick Jimenez Berumen, Mikhail Bessmeltsev, and Edward Chien. 2023. Singularity-Free Frame Fields for Line Drawing Vectorization. Computer Graphics Forum 42, 5 (2023), e14901. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14901
[21]
Tobias Hinz, Matthew Fisher, Oliver Wang, Eli Shechtman, and Stefan Wermter. 2022. CharacterGAN: Few-Shot Keypoint Character Animation and Reposing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1988--1997.
[22]
Berthold K.P. Horn and Brian G. Schunck. 1981. Determining Optical Flow. Artificial Intelligence 17, 1 (1981), 185--203.
[23]
Zhewei Huang, Ailin Huang, Xiaotao Hu, Chen Hu, Jun Xu, and Shuchang Zhou. 2024. Scale-Adaptive Feature Aggregation for Efficient Space-Time Video Super-Resolution. In Winter Conference on Applications of Computer Vision (WACV).
[24]
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. 2022. Real-Time Intermediate Flow Estimation for Video Frame Interpolation. In Proceedings of the European Conference on Computer Vision (ECCV).
[25]
Alec Jacobson, Ilya Baran, Jovan Popović, and Olga Sorkine. 2011. Bounded Biharmonic Weights for Real-Time Deformation. ACM Trans. Graph. 30, 4, Article 78 (jul 2011), 8 pages.
[26]
Eakta Jain, Yaser Sheikh, Moshe Mahler, and Jessica Hodgins. 2012. Three-dimensional proxies for hand-drawn characters. ACM Trans. Graph. 31, 1, Article 8 (Feb. 2012), 16 pages.
[27]
Jie Jiang, Hock Soon Seah, and Hong Ze Liew. 2022. Stroke-Based Drawing and Inbetweening with Boundary Strokes. Computer Graphics Forum 41, 1 (2022), 257--269. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14433
[28]
Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran. 2023. FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2070--2081.
[29]
Olga A. Karpenko and John F. Hughes. 2006. SmoothSketch: 3D free-form shapes from complex sketches. ACM Trans. Graph. 25, 3 (jul 2006), 589--598.
[30]
Alan Kitching. 1977. Antics---Graphic animation by computer. Computers & Graphics 2, 4 (1977), 219--223.
[31]
Jan J. Koenderink and Andrea J. van Doorn. 1982. The Shape of Smooth Objects and the Way Contours End. Perception 11, 2 (1982), 129--137. arXiv:https://doi.org/10.1068/p110129 7155766.
[32]
Alexander Kort. 2002. Computer Aided Inbetweening. In Proceedings of the 2nd International Symposium on Non-Photorealistic Animation and Rendering (Annecy, France) (NPAR '02). Association for Computing Machinery, New York, NY, USA, 125--132.
[33]
Shahar Z. Kovalsky, Meirav Galun, and Yaron Lipman. 2016. Accelerated Quadratic Proxy for Geometric Optimization. ACM Trans. Graph. 35, 4, Article 134 (jul 2016), 11 pages.
[34]
Minchen Li, Danny M. Kaufman, Vladimir G. Kim, Justin Solomon, and Alla Sheffer. 2018. OptCuts: Joint Optimization of Surface Cuts and Parameterization. ACM Transactions on Graphics 37, 6 (2018).
[35]
Xiaoyu Li, Bo Zhang, Jing Liao, and Pedro V. Sander. 2022. Deep Sketch-Guided Cartoon Video Inbetweening. IEEE Transactions on Visualization and Computer Graphics 28, 8 (aug 2022), 2938--2952.
[36]
Dongquan Liu, Quan Chen, Jun Yu, Huiqin Gu, Dacheng Tao, and Hock Soon Seah. 2011. Stroke Correspondence Construction Using Manifold Learning. Computer Graphics Forum 30, 8 (2011), 2194--2207. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2011.01969.x
[37]
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (Oct. 2015), 248:1--248:16.
[38]
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations. https://openreview.net/forum?id=Bkg6RiCqY7
[39]
James McCann and Nancy Pollard. 2009. Local Layering. In ACM SIGGRAPH 2009 Papers (New Orleans, Louisiana) (SIGGRAPH '09). Association for Computing Machinery, New York, NY, USA, Article 84, 7 pages.
[40]
R. Miyauchi, T. Fukusato, H. Xie, and K. Miyata. 2021a. Stroke Correspondence by Labeling Closed Areas. In 2021 Nicograph International (NicoInt). IEEE Computer Society, Los Alamitos, CA, USA, 34--41.
[41]
Ryoma Miyauchi, Yichen Peng, Tsukasa Fukusato, and Haoran Xie. 2021b. Skeleton2Stroke: Interactive Stroke Correspondence Editing with Pose Features. In SIGGRAPH Asia 2021 Technical Communications (Tokyo, Japan) (SA '21). Association for Computing Machinery, New York, NY, USA, Article 6, 4 pages.
[42]
Haoran Mo, Chengying Gao, and Ruomei Wang. 2024. Joint Stroke Tracing and Correspondence for 2D Animation. ACM Transactions on Graphics (2024).
[43]
Mariia Myronova, William Neveu, and Mikhail Bessmeltsev. 2023. Differential Operators on Sketches via Alpha Contours. ACM Trans. Graph. 42, 4, Article 69 (jul 2023), 15 pages.
[44]
Simon Niklaus and Feng Liu. 2020. Softmax Splatting for Video Frame Interpolation. In IEEE Conference on Computer Vision and Pattern Recognition.
[45]
Jorge Nocedal and Stephen J. Wright (Eds.). 1999. Numerical Optimization. SpringerVerlag, New York.
[46]
G. Noris, D. Sýkora, S. Coros, B. Whited, M. Simmons, A. Hornung, M. Gross, and R. W. Sumner. 2011. Temporal Noise Control for Sketchy Animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Non-Photorealistic Animation and Rendering (Vancouver, British Columbia, Canada) (NPAR '11). Association for Computing Machinery, New York, NY, USA, 93--98.
[47]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. 2024. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research (2024). https://openreview.net/forum?id=a68SUt6zFt
[48]
Ulrich Pinkall and Konrad Polthier. 1993. Computing Discrete Minimal Surfaces and Their Conjugates. Experimental Mathematics 2, 1 (1993), 15--36. arXiv:https://doi.org/10.1080/10586458.1993.10504266
[49]
Omid Poursaeed, Vladimir Kim, Eli Shechtman, Jun Saito, and Serge Belongie. 2020. Neural puppet: Generative layered cartoon characters. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3346--3356.
[50]
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. 2022. FILM: Frame Interpolation for Large Motion. In European Conference on Computer Vision (ECCV).
[51]
William T. Reeves. 1981. Inbetweening for Computer Animation Utilizing Moving Point Constraints. SIGGRAPH Comput. Graph. 15, 3 (aug 1981), 263--269.
[52]
Alec Rivers, Takeo Igarashi, and Frédo Durand. 2010. 2.5D Cartoon Models. ACM Trans. Graph. 29, 4, Article 59 (jul 2010), 7 pages.
[53]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. CoRR abs/1505.04597 (2015). http://dblp.uni-trier.de/db/journals/corr/corr1505.html#RonnebergerFB15
[54]
Jonathan Richard Shewchuk. 1996. Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator. In Selected Papers from the Workshop on Applied Computational Geormetry, Towards Geometric Engineering (FCRC '96/WACG '96). Springer-Verlag, Berlin, Heidelberg, 203--222.
[55]
Li Siyao, Tianpei Gu, Weiye Xiao, Henghui Ding, Ziwei Liu, and Chen Change Loy. 2023. Deep Geometrized Cartoon Line Inbetweening. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7291--7300.
[56]
Li Siyao, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris Metaxas, Chen Change Loy, and Ziwei Liu. 2021. Deep Animation Video Interpolation in the Wild. In CVPR.
[57]
J Smith and S Schaefer. 2015. Bijective Parameterization with Free Boundaries. Acm Transactions on Graphics 34, 4 (2015), 9.
[58]
Olga Sorkine and Marc Alexa. 2007. As-Rigid-As-Possible Surface Modeling. Proceedings of the fifth Eurographics symposium on Geometry processing (2007), 109--116. ISBN: 9783905673463.
[59]
Paul Starke, Sebastian Starke, Taku Komura, and Frank Steinicke. 2023. Motion In-Betweening with Phase Manifolds. Proc. ACM Comput. Graph. Interact. Tech. 6, 3, Article 37 (aug 2023), 17 pages.
[60]
Fleischer Studios. 1939. Aladdin and His Wonderful Lamp. https://archive.org/details/popeye-meets-aladdin.
[61]
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. 2021. Resolution-robust Large Mask Inpainting with Fourier Convolutions. arXiv preprint arXiv.2109.07161 (2021).
[62]
Daniel Sýkora, John Dingliana, and Steven Collins. 2009. As-Rigid-as-Possible Image Registration for Hand-Drawn Cartoon Animations. In Proceedings of the 7th International Symposium on Non-Photorealistic Animation and Rendering (New Orleans, Louisiana) (NPAR '09). Association for Computing Machinery, New York, NY, USA, 25--33.
[63]
Daniel Sýkora, Ladislav Kavan, Martin Čadík, Ondřej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-Ray: BasRelief Meshes for Adding Global Illumination Effects to Hand-Drawn Characters. ACM Transaction on Graphics 33, 2 (2014), 16.
[64]
D. Sýkora, D. Sedlacek, S. Jinchao, J. Dingliana, and S. Collins. 2010. Adding Depth to Cartoons Using Sparse Depth (In)equalities. Computer Graphics Forum 29, 2 (2010), 615--623. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2009.01631.x
[65]
Shuhei Watanabe. 2023. Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance. arXiv:2304.11127 [cs.LG]
[66]
Brian Whited, Gioacchino Noris, Maryann Simmons, Robert W. Sumner, Markus Gross, and Jarek Rossignac. 2010. BetweenIT: An Interactive Tool for Tight Inbetweening. Computer Graphics Forum 29, 2 (2010), 605--614. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2009.01630.x
[67]
Richard Williams. 2001. Animators Survival Kit: A Working Manual Of Methods Principles And Formulas For Computer (7th printing edition ed.). Faber & Faber, London.
[68]
Wenwu Yang, Hock-Soon Seah, Quan Chen, Hong-Ze Liew, and Daniel Sýkora. 2018. FTP-SC: Fuzzy Topology Preserving Stroke Correspondence. Computer Graphics Forum 37, 8 (2018), 125--135.
[69]
Jun Yu, Wei Bian, Mingli Song, Jun Cheng, and Dacheng Tao. 2012. Graph based transductive learning for cartoon correspondence construction. Neurocomputing 79 (2012), 105--114.
[70]
Zhiyang Yu, Yu Zhang, Xujie Xiang, Dongqing Zou, Xijun Chen, and Jimmy S Ren. 2022. Deep Bayesian Video Frame Interpolation. In European Conference on Computer Vision. Springer, 144--160.
[71]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.
[72]
Haichao Zhu, Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2016. Globally Optimal Toon Tracking. ACM Transactions on Graphics (SIGGRAPH 2016 issue) 35, 4 (July 2016), 75:1--75:10.
[73]
Yufeng Zhu, Jovan Popović, Robert Bridson, and Danny M. Kaufman. 2017. Planar Interpolation with Extreme Deformation, Topology Change and Dynamics. ACM Trans. Graph. 36, 6, Article 213 (nov 2017), 15 pages.
Index Terms
- Skeleton-Driven Inbetweening of Bitmap Character Drawings
Recommendations
A Method for Animating Children’s Drawings of the Human Figure
Children’s drawings have a wonderful inventiveness, creativity, and variety to them. We present a system that automatically animates children’s drawings of the human figure, is robust to the variance inherent in these depictions, and is simple and ...
Secondary Motion for Performed 2D Animation
UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and TechnologyWhen bringing animated characters to life, artists often augment the primary motion of a figure by adding secondary animation -- subtle movement of parts like hair, foliage or cloth that complements and emphasizes the primary motion. Traditionally, ...
Frame Skeleton Based Auto-Inbetweening in Computer Assisted Cel Animation
CW '04: Proceedings of the 2004 International Conference on CyberworldsAutomatic inbetweening is one of the main focuses in computer assisted cel animation as inbetweening process is traditionally time-consuming and labor-intensive in the production of cartoon animation. Most previous work demand either lots of modeling or ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 November 2024
Published in TOG Volume 43, Issue 6
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- NSERC
- FRQNT NOVA
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 77Total Downloads
- Downloads (Last 12 months)77
- Downloads (Last 6 weeks)23
Reflects downloads up to 07 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in