skip to main content
article

Defocus video matting

Published: 01 July 2005 Publication History

Abstract

Video matting is the process of pulling a high-quality alpha matte and foreground from a video sequence. Current techniques require either a known background (e.g., a blue screen) or extensive user interaction (e.g., to specify known foreground and background elements). The matting problem is generally under-constrained, since not enough information has been collected at capture time. We propose a novel, fully autonomous method for pulling a matte using multiple synchronized video streams that share a point of view but differ in their plane of focus. The solution is obtained by directly minimizing the error in filter-based image formation equations, which are over-constrained by our rich data stream. Our system solves the fully dynamic video matting problem without user assistance: both the foreground and background may be high frequency and have dynamic content, the foreground may resemble the background, and the scene is lit by natural (as opposed to polarized or collimated) illumination.

Supplementary Material

MP4 File (pps021.mp4)

References

[1]
Apostoloff, N. E., and Fitzgibbon, A. W. 2004. Bayesian video matting using learnt image priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 407--414.
[2]
Asada, N., Fujiwara, H., and Matsuyama, T. 1998. Seeing behind the scene: analysis of photometric properties of occluding edges by the reversed projection blurring model. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 2, 155--67.
[3]
Ben-Ezra, M., and Nayar, S. 2004. Jitter camera: High resolution video from a low resolution detector. In IEEE CVPR, 135--142.
[4]
Bhasin, S. S., and Chaudhuri, S. 2001. Depth from defocus in presence of partial self occlusion. Proceedings of the International Conference on Computer Vision 1, 2, 488--93.
[5]
Blake, A., Rother, C., Brown, M., Perez. P., and Torr, P. 2004. Interactive image segmentation using an adaptive gmmrf model. Proceedings of the European Conference on Computer Vision (ECCV).
[6]
Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proceedings of IEEE CVPR 2001, IEEE Computer Society, vol. 2, 264--271.
[7]
Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Trans. on Graphics 21, 3 (July), 243--248.
[8]
Debevec, P. E., and Malik, J. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 369--378.
[9]
Favaro, P., and Soatto, S. 2003. Seeing beyond occlusions (and other marvels of a finite lens aperture). In IEEE CVPR, 579--586.
[10]
Fleischer, M., 1917. Method of producing moving picture cartoons. US Patent no. 1,242,674.
[11]
Glassner, A. S. 1995. Principles of Digital Image Synthesis. Morgan Kaufmann Publishers, Inc.
[12]
Haralick, R. M., Sternberg, S. R., and Zhuang, X. 1987. Image analysis using mathematical morphology. IEEE PAMI 9, 4, 532--550.
[13]
Hecht, E. 1998. Optics Third Edition. Addison Wesley Longman, Inc.
[14]
Hillman, P., Hannah, J., and Renshaw, D. 2001. Alpha channel estimation in high resolution images and image sequences. In Proceedings of IEEE CVPR 2001, IEEE Computer Society, vol. 1, 1063--1068.
[15]
Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, I., and Bolas, M. 2004. Synthetic aperture confocal imaging. ACM Trans. Graph. 23, 3, 825--834.
[16]
Malvar, H. S., Wei He, L., and Cutler, R. 2004. High-quality linear interpolation for demosaicing of bayer-patterned color images. Proceedings of the IEEE International Conference on Speech, Acoustics, and Signal Processing.
[17]
Nayar, S. K., and Branzoi, V. 2003. Adaptive dynamic range imaging: Optical control of pixel exposures over space and time. In Proceedings of the International Conference on Computer Vision (ICCV), 1168--1175.
[18]
Nayar, S. K., Watanabe, M., and Noguchi, M. 1996. Real-time focus range sensor. IEEE PAMI 18, 12, 1186--1198.
[19]
Nocedal, J., and Wright, S. J. 1999. Numerical Optimization. Springer Verlag.
[20]
Pentland, A. P. 1987. A new sense for depth of field. IEEE PAMI 9, 4, 523--531.
[21]
Porter, T., and Duff, T. 1984. Compositing digital images. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques, ACM Press, 253--259.
[22]
Potmesil, M., and Chakravarty, I. 1983. Modeling motion blur in computer-generated images. Computer Graphics 17, 3 (July), 389--399.
[23]
Rother, C., Kolmogorov, V., and Blake, A. 2004. "grabcut": interactive foreground extraction using iterated graph cuts. ACM Trans. on Graphics 23, 3, 309--314.
[24]
Ruzon, M. A., and Tomasi, C. 2000. Alpha estimation in natural images. In CVPR 2000, vol. 1, 18--25.
[25]
Schechner, Y. Y., Kiryati, N., and Basri, R. 2000. Separation of transparent layers using focus. International Journal of Computer Vision, 25--39.
[26]
Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM Press, 259--268.
[27]
Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Transactions on Graphics (August), 315--321.
[28]
Yahav, G., and Iddan, G. 2002. 3dv systems' zcam. Broadcast Engineering.
[29]
Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. on Graphics 23, 3, 600--608.

Cited By

View all
  • (2023)Color-aware Deep Temporal Backdrop Duplex Matting SystemProceedings of the 14th Conference on ACM Multimedia Systems10.1145/3587819.3590973(205-216)Online publication date: 7-Jun-2023
  • (2023)Lens Parameter Estimation for Realistic Depth of Field Modeling2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00052(499-508)Online publication date: 1-Oct-2023
  • (2023)Ultrahigh Resolution Image/Video Matting with Spatio-Temporal Sparsity2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01356(14112-14121)Online publication date: Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 24, Issue 3
July 2005
826 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/1073204
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2005
Published in TOG Volume 24, Issue 3

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)17
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Color-aware Deep Temporal Backdrop Duplex Matting SystemProceedings of the 14th Conference on ACM Multimedia Systems10.1145/3587819.3590973(205-216)Online publication date: 7-Jun-2023
  • (2023)Lens Parameter Estimation for Realistic Depth of Field Modeling2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00052(499-508)Online publication date: 1-Oct-2023
  • (2023)Ultrahigh Resolution Image/Video Matting with Spatio-Temporal Sparsity2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01356(14112-14121)Online publication date: Jun-2023
  • (2021)Research on multi-sensor high dynamic range imaging technology and applicationFirst Optics Frontier Conference10.1117/12.2599254(15)Online publication date: 18-Jun-2021
  • (2021)Joint Depth and Defocus Estimation From a Single Image Using Physical ConsistencyIEEE Transactions on Image Processing10.1109/TIP.2021.306190130(3419-3433)Online publication date: 1-Jan-2021
  • (2021)Deep Video Matting via Spatio-Temporal Alignment and Aggregation2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR46437.2021.00690(6971-6980)Online publication date: Jun-2021
  • (2021)Matte ExtractionComputer Vision10.1007/978-3-030-63416-2_12(795-799)Online publication date: 13-Oct-2021
  • (2020)Unsupervised Video Matting via Sparse and Low-Rank RepresentationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2019.289533142:6(1501-1514)Online publication date: 1-Jun-2020
  • (2019)Image Composition of Partially Occluded ObjectsComputer Graphics Forum10.1111/cgf.1386738:7(641-650)Online publication date: 14-Nov-2019
  • (2019)Illumination-Guided Video Composition via Gradient Consistency OptimizationIEEE Transactions on Image Processing10.1109/TIP.2019.291676928:10(5077-5090)Online publication date: 1-Oct-2019
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media