Michal Kučera
ABSTRACT
In this paper we present StyleBin—an approach to example-based stylization of videos that can produce consistent binocular depiction of stylized content on stereoscopic displays. Given the target sequence and a set of stylized keyframes accompanied by information about depth in the scene, we formulate an optimization problem that converts the target video into a pair of stylized sequences, in which each frame consists of a set of seamlessly stitched patches taken from the original stylized keyframe. The aim of the optimization process is to align the individual patches so that they respect the semantics of the given target scene, while at the same time also following the prescribed local disparity in the corresponding viewpoints and being consistent in time. In contrast to previous depth-aware style transfer techniques, our approach is the first that can deliver semantically meaningful stylization and preserve essential visual characteristics of the given artistic media. We demonstrate the practical utility of the proposed method in various stylization use cases.
Supplemental Material
- Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24.Google Scholar
Digital Library
- Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation By Example. ACM Transactions on Graphics 32, 4 (2013), 119.Google ScholarDigital Library
- Dennis R. Bukenberger, Katharina Schwarz, and Hendrik P. A. Lensch. 2018. Stereo-Consistent Contours in Object Space. Computer Graphics Forum 37, 1 (2018), 301–312.Google ScholarCross Ref
- Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2018. Stereoscopic Neural Style Transfer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 6654–6663.Google ScholarCross Ref
- Dónal Egan, Martin Alain, and Aljosa Smolic. 2021. Light Field Style Transfer with Local Angular Consistency. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. 2300–2304.Google ScholarCross Ref
- Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sýkora. 2016. StyLit: Illumination-Guided Example-Based Stylization of 3D Renderings. ACM Transactions on Graphics 35, 4 (2016), 92.Google ScholarDigital Library
- Jakub Fišer, Ondřej Jamriška, David Simons, Eli Shechtman, Jingwan Lu, Paul Asente, Michal Lukáč, and Daniel Sýkora. 2017. Example-Based Synthesis of Stylized Facial Animations. ACM Transactions on Graphics 36, 4 (2017), 155.Google ScholarDigital Library
- David Futschik, Michal Kučera, Michal Lukáč, Zhaowen Wang, Eli Shechtman, and Daniel Sýkora. 2021. STALP: Style Transfer with Auxiliary Limited Pairing. Computer Graphics Forum 40, 2 (2021), 563–573.Google ScholarCross Ref
- Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2414–2423.Google ScholarCross Ref
- Xinyu Gong, Haozhi Huang, Lin Ma, Fumin Shen, Wei Liu, and Tong Zhang. 2018. Neural Stereoscopic Image Style Transfer. In Proceedings of European Conference on Computer Vision. 56–71.Google ScholarDigital Library
- Ivan Grishchenko, Artsiom Ablavatski, Yury Kartynnik, Karthik Raveendran, and Matthias Grundmann. 2020. Attention Mesh: High-fidelity Face Mesh Prediction in Real-time. In Proceedings of the CVPR Workshop on Computer Vision for Augmented and Virtual Reality.Google Scholar
- Filip Hauptfleisch, Ondřej Texler, Aneta Texler, Jaroslav Křivánek, and Daniel Sýkora. 2020. StyleProp: Real-time Example-based Stylization of 3D Models. Computer Graphics Forum 39, 7 (2020), 575–586.Google ScholarCross Ref
- Hsin-Ping Huang, Hung-Yu Tseng, Saurabh Saini, Maneesh Singh, and Ming-Hsuan Yang. 2021. Learning to Stylize Novel Views. In Proceedings of IEEE International Conference on Computer Vision. 13869–13878.Google ScholarCross Ref
- Lesley Istead and Craig S. Kaplan. 2018. Stylized Stereoscopic 3D Line Drawings from 3D Images. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 20.Google ScholarDigital Library
- Lesley Istead, Andreea Pocol, Craig S. Kaplan, Isaac Watt, Nick Lemoing, and Alicia Yang. 2021. Generating Rough Stereoscopic 3D Line Drawings from 3D Images. In Proceedings of Graphics Interface. 178–185.Google Scholar
- Ondřej Jamriška, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. LazyFluids: Appearance Transfer for Fluid Animations. ACM Transactions on Graphics 34, 4 (2015), 92.Google ScholarDigital Library
- Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2019. Stylizing Video by Example. ACM Transactions on Graphics 38, 4 (2019), 107.Google ScholarDigital Library
- Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015. Self Tuning Texture Optimization. Computer Graphics Forum 34, 2 (2015), 349–360.Google ScholarDigital Library
- Yongjin Kim, Yunjin Lee, Henry Kang, and Seungyong Lee. 2013. Stereoscopic 3D Line Drawing. ACM Transactions on Graphics 32, 4 (2013), 57.Google ScholarDigital Library
- Nicholas I. Kolkin, Jason Salavon, and Gregory Shakhnarovich. 2019. Style Transfer by Relaxed Optimal Transport and Self-Similarity. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 10051–10060.Google ScholarCross Ref
- Till Kroeger, Radu Timofte, Dengxin Dai, and Luc Van Gool. 2016. Fast Optical Flow Using Dense Inverse Search. In Proceedings of European Conference on Computer Vision. 471–488.Google ScholarCross Ref
- Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual Attribute Transfer Through Deep Image Analogy. ACM Transactions on Graphics 36, 4 (2017), 120.Google ScholarDigital Library
- Sheng-Jie Luo, Ying-Tse Sun, I-Chao Shen, Bing-Yu Chen, and Yung-Yu Chuang. 2015. Geometrically Consistent Stereoscopic Image Editing Using Patch-Based Synthesis. IEEE Transactions on Visualization and Computer Graphics 21, 1(2015), 56–67.Google ScholarCross Ref
- S. Mahdi H. Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yağız Aksoy. 2021. Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 9685–9694.Google ScholarCross Ref
- Lesley Northam, Paul Asente, and Craig S. Kaplan. 2012. Consistent Stylization and Painterly Rendering of Stereoscopic 3D Images. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 47–56.Google Scholar
- Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson Image Editing. ACM Transactions on Graphics 22, 3 (2003), 313–318.Google ScholarDigital Library
- Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. 2018. Artistic Style Transfer for Videos and Spherical Images. International Journal of Computer Vision 126, 11 (2018), 1199–1219.Google ScholarDigital Library
- Efstathios Stavrakis and Margrit Gelautz. 2004. Image-Based Stereoscopic Painterly Rendering. In Proceedings of the Eurographics Conference on Rendering Techniques. 53–60.Google Scholar
- Daniel Sýkora, Ondřej Jamriška, Ondřej Texler, Jakub Fišer, Michal Lukáč, Jingwan Lu, and Eli Shechtman. 2019. StyleBlit: Fast Example-Based Stylization with Local Guidance. Computer Graphics Forum 38, 2 (2019), 83–91.Google ScholarCross Ref
- Krzysztof Templin, Piotr Didyk, Karol Myszkowski, and Hans-Peter Seidel. 2014. Perceptually-motivated Stereoscopic Film Grain. Computer Graphics Forum 33, 7 (2014), 349–358.Google ScholarDigital Library
- Ondřej Texler, David Futschik, Michal Kučera, Ondřej Jamriška, Šárka Sochorová, Menglei Chai, Sergey Tulyakov, and Daniel Sýkora. 2020. Interactive Video Stylization Using Few-Shot Patch-Based Training. ACM Transactions on Graphics 39, 4 (2020), 73.Google ScholarDigital Library
- Liang Wang, Hailin Jin, Ruigang Yang, and Minglun Gong. 2008. Stereoscopic inpainting: Joint color and depth completion from stereo images. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-Time Completion of Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3(2007), 463–476.Google ScholarDigital Library
Index Terms
- StyleBin: Stylizing Video by Example in Stereo
Recommendations
Stylizing video by example
We introduce a new example-based approach to video stylization, with a focus on preserving the visual quality of the style, user controllability and applicability to arbitrary video. Our method gets as input one or more keyframes that the artist chooses ...
Stylizing animation by example
Skilled artists, using traditional media or modern computer painting tools, can create a variety of expressive styles that are very appealing in still images, but have been unsuitable for animation. The key difficulty is that existing techniques lack ...
Stereo in post-production
3DVP '10: Proceedings of the 1st international workshop on 3D video processingStereo film production has recently made a huge impact with the success of James Cameron's 3D movie Avatar. Stereo presents a series of challenges: from the construction and calibration of stereo capture rigs, the subsequent work flow, to the post-...
Comments