skip to main content
research-article

Interactive High-Quality Green-Screen Keying via Color Unmixing

Published: 08 August 2016 Publication History

Abstract

Due to the widespread use of compositing in contemporary feature films, green-screen keying has become an essential part of postproduction workflows. To comply with the ever-increasing quality requirements of the industry, specialized compositing artists spend countless hours using multiple commercial software tools, while eventually having to resort to manual painting because of the many shortcomings of these tools. Due to the sheer amount of manual labor involved in the process, new green-screen keying approaches that produce better keying results with less user interaction are welcome additions to the compositing artist’s arsenal. We found that—contrary to the common belief in the research community—production-quality green-screen keying is still an unresolved problem with its unique challenges. In this article, we propose a novel green-screen keying method utilizing a new energy minimization-based color unmixing algorithm. We present comprehensive comparisons with commercial software packages and relevant methods in literature, which show that the quality of our results is superior to any other currently available green-screen keying solution. It is important to note that, using the proposed method, these high-quality results can be generated using only one-tenth of the manual editing time that a professional compositing artist requires to process the same content having all previous state-of-the-art tools at one’s disposal.

Supplementary Material

aksoy (aksoy.zip)
Supplemental movie, appendix, image and software files for, Interactive High-Quality Green-Screen Keying via Color Unmixing
MP4 File (tog-09.mp4)

References

[1]
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 11, 2274--2282.
[2]
Xue Bai, Jue Wang, and David Simons. 2011. Towards temporally-coherent video matting. In Proceedings of MIRAGE.
[3]
Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro. 2009. Video SnapCut: Robust video object cutout using localized classifiers. ACM Transactions on Graphics 28, 3, 70:1--70:11.
[4]
Dimitri P. Bertsekas. 1982. The method of multipliers for equality constrained problems. In Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York, NY, 96--157.
[5]
Y. Boykov and V. Kolmogorov. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 9, 1124--1137.
[6]
Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11, 1222--1239.
[7]
Robert Carroll, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Transactions on Graphics 30, 4, 43:1--43:10.
[8]
Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Transactions on Graphics 34, 4, 139:1--139:11.
[9]
Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. 2013a. KNN matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 9, 2175--2188.
[10]
Xiaowu Chen, Dongqing Zou, S. Z. Zhou, Qinping Zhao, and Ping Tan. 2013b. Image matting with local and nonlocal smooth priors. In Proceedings of CVPR.
[11]
Inchang Choi, Minhaeng Lee, and Yu-Wing Tai. 2012. Video matting using multi-frame nonlocal matting Laplacian. In Proceedings of ECCV.
[12]
Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David H. Salesin, and Richard Szeliski. 2002. Video matting of complex scenes. Transactions on Graphics 243--248.
[13]
Yung-Yu Chuang, Brian Curless, David H. Salesin, and Richard Szeliski. 2001. A Bayesian approach to digital matting. In Proceedings of CVPR.
[14]
Jialue Fan, Xiaohui Shen, and Ying Wu. 2012. Scribble tracker: A matting-based approach for robust tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 8, 1633--1644.
[15]
Eduardo S. L. Gastal and Manuel M. Oliveira. 2010. Shared sampling for real-time alpha matting. Computer Graphics Forum 29, 2, 575--584.
[16]
Anselm Grundhöfer and Oliver Bimber. 2008. VirtualStudio2Go: Digital video composition for real environments. ACM Transactions on Graphics 27, 5, 151:1--151:8.
[17]
Anselm Grundhöfer, Daniel Kurz, Sebastian Thiele, and Oliver Bimber. 2010. Color invariant chroma keying and color spill neutralization for dynamic scenes and cameras. The Visual Computer 26, 9, 1167--1176.
[18]
Bei He, Guijin Wang, Chenbo Shi, Xuanwu Yin, Bo Liu, and Xinggang Lin. 2013. Iterative transductive learning for alpha matting. In Proceedings of ICIP.
[19]
Kaiming He, C. Rhemann, C. Rother, Xiaoou Tang, and Jian Sun. 2011. A global sampling method for alpha matting. In Proceedings of CVPR.
[20]
Meiguang Jin, Byoung-Kwang Kim, and Woo-Jin Song. 2014. Adaptive propagation-based color-sampling for alpha matting. IEEE Transactions on Circuits and Systems for Video Technology 24, 7, 1101--1110.
[21]
Jubin Johnson, Deepu Rajan, and Hisham Cholakkal. 2014. Sparse codes as alpha matte. In Proceedings of BMVC.
[22]
V. Kolmogorov and R. Zabih. 2004. What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 2, 147--159.
[23]
P. Lee and Ying Wu. 2011. Nonlocal matting. In Proceedings of CVPR.
[24]
Anat Levin, Dani Lischinski, and Yair Weiss. 2008a. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2, 228--242.
[25]
Anat Levin, Alex Rav-Acha, and Dani Lischinski. 2008b. Spectral matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 10, 1699--1712.
[26]
Dingzeyu Li, Qifeng Chen, and Chi-Keung Tang. 2013. Motion-aware KNN Laplacian for video matting. In Proceedings of ICCV.
[27]
Yin Li, Jian Sun, and Heung-Yeung Shum. 2005. Video object cut and paste. ACM Transactions on Graphics 24, 3, 595--600.
[28]
Hai Ting Lin, Yu-Wing Tai, and M. S. Brown. 2011. Motion regularization for matting motion blurred objects. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 11, 2329--2336.
[29]
Christoph Rhemann, Carsten Rother, and Margrit Gelautz. 2008. Improving color modeling for alpha matting. In Proceedings of BMVC.
[30]
Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A perceptually motivated online benchmark for image matting. In Proceedings of CVPR.
[31]
Mark A. Ruzon and Carlo Tomasi. 2000. Alpha estimation in natural images. In Proceedings of CVPR.
[32]
E. Shahrian, B. Price, S. Cohen, and D. Rajan. 2014. Temporally coherent and spatially accurate video matting. Computer Graphics Forum 33, 2, 381--390.
[33]
E. Shahrian and D. Rajan. 2012. Weighted color and texture sample selection for image matting. In Proceedings of CVPR.
[34]
E. Shahrian, D. Rajan, B. Price, and S. Cohen. 2013. Improving image matting using comprehensive sampling sets. In Proceedings of CVPR.
[35]
Yongfang Shi, O. C. Au, Jiahao Pang, K. Tang, Wenxiu Sun, Hong Zhang, Wenjing Zhu, and Luheng Jia. 2013. Color clustering matting. In Proceedings of ICME.
[36]
YiChang Shih, Dilip Krishnan, Fredo Durand, and William T. Freeman. 2015. Reflection removal using ghosting cues. In Proceedings of CVPR.
[37]
D. Singaraju, C. Rother, and C. Rhemann. 2009. New appearance models for natural image matting. In Proceedings of CVPR.
[38]
D. Singaraju and R. Vidal. 2011. Estimation of alpha mattes for multiple image layers. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 7, 1295--1309.
[39]
Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based rendering for scenes with reflections. ACM Transactions on Graphics 31, 4, 100:1--100:10.
[40]
Alvy Ray Smith and James F. Blinn. 1996. Blue screen matting. ACM Transactions on Graphics (1996), 259--268.
[41]
Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeung Shum. 2004. Poisson matting. ACM Transactions on Graphics 23, 3, 315--321.
[42]
Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Proceedings of CVPR.
[43]
Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2007. Soft color segmentation and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 9, 1520--1537.
[44]
Zhen Tang, Zhenjiang Miao, Yanli Wan, and Dianyong Zhang. 2012. Video matting via opacity propagation. The Visual Computer 28, 1, 47--61.
[45]
Jue Wang, Pravin Bhat, R. Alex Colburn, Maneesh Agrawala, and Michael F. Cohen. 2005. Interactive video cutout. ACM Transactions on Graphics 24, 3, 585--594.
[46]
Jue Wang and M. F. Cohen. 2005. An iterative optimization approach for unified image segmentation and matting. In Proceedings of ICCV.
[47]
Jue Wang and M. F. Cohen. 2007. Optimized color sampling for robust matting. In Proceedings of CVPR.
[48]
Tai-Pang Wu, Chi-Keung Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. Natural shadow matting. ACM Transactions on Graphics 26, 2.
[49]
Sai-Kit Yeung, Tai-Pang Wu, and Chi-Keung Tang. 2008. Extracting smooth and transparent layers from a single image. In Proceedings of CVPR.
[50]
Fan Zhong, Xueying Qin, Qunsheng Peng, and Xiangxu Meng. 2012. Discontinuity-aware video object cutout. ACM Transactions on Graphics 31, 6, 175:1--175:10.

Cited By

View all
  • (2024)Transparent Image Layer Diffusion using Latent TransparencyACM Transactions on Graphics10.1145/365815043:4(1-15)Online publication date: 19-Jul-2024
  • (2024)Polar Matte: Fully Computational Ground-Truth-Quality Alpha Matte Extraction for Images and Video using Polarized Screen Matting2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00374(3901-3909)Online publication date: 16-Jun-2024
  • (2023)ColorfulCurves: Palette-Aware Lightness Control and Color Editing via Sparse OptimizationACM Transactions on Graphics10.1145/359240542:4(1-12)Online publication date: 26-Jul-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 35, Issue 5
September 2016
156 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/2965650
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 August 2016
Accepted: 01 March 2016
Revised: 01 February 2016
Received: 01 September 2015
Published in TOG Volume 35, Issue 5

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Green-screen keying
  2. image matting
  3. interactive segmentation
  4. soft segmentation

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)17
  • Downloads (Last 6 weeks)1
Reflects downloads up to 01 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Transparent Image Layer Diffusion using Latent TransparencyACM Transactions on Graphics10.1145/365815043:4(1-15)Online publication date: 19-Jul-2024
  • (2024)Polar Matte: Fully Computational Ground-Truth-Quality Alpha Matte Extraction for Images and Video using Polarized Screen Matting2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00374(3901-3909)Online publication date: 16-Jun-2024
  • (2023)ColorfulCurves: Palette-Aware Lightness Control and Color Editing via Sparse OptimizationACM Transactions on Graphics10.1145/359240542:4(1-12)Online publication date: 26-Jul-2023
  • (2023) LoCoPalettes : Local Control for Palette‐based Image Editing Computer Graphics Forum10.1111/cgf.1489242:4Online publication date: 26-Jul-2023
  • (2023)Effective Eyebrow Matting with Domain AdaptationComputer Graphics Forum10.1111/cgf.1468241:7(347-358)Online publication date: 20-Mar-2023
  • (2022)Fast Nonlinear Image Unblending2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV51458.2022.00325(3193-3201)Online publication date: Jan-2022
  • (2022)A Blind Color Separation Model for Faithful Palette-Based Image RecoloringIEEE Transactions on Multimedia10.1109/TMM.2021.306746324(1545-1557)Online publication date: 1-Jan-2022
  • (2022)Human Instance Matting via Mutual Guidance and Multi-Instance Refinement2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00267(2637-2646)Online publication date: Jun-2022
  • (2021)Automatic, Illumination-Invariant and Real-Time Green-Screen Keying Using Deeply Guided Linear ModelsSymmetry10.3390/sym1308145413:8(1454)Online publication date: 9-Aug-2021
  • (2021)Real-time Global Illumination Decomposition of VideosACM Transactions on Graphics10.1145/337475340:3(1-16)Online publication date: 10-Aug-2021
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media