skip to main content
10.1145/2824840.2824845acmotherconferencesArticle/Chapter ViewAbstractPublication PagescvmpConference Proceedingsconference-collections
research-article

Multi-camera system for depth based visual effects and compositing

Published: 24 November 2015 Publication History

Abstract

Post-production technologies like visual effects (VFX) or digital grading are essential for visual storytelling in today's movie production. Tremendous efforts are made to seamlessly integrate computer generated (CG) elements into live action plates, to refine looks and to provide various delivery formats such as Stereo 3D (S3D). Thus, additional tools to assist and improve element integration, grading and S3D techniques could ease complicated, time consuming, manual and costly processes. Geometric data like depth information is a key for these tasks but a typical main unit camera shoot does not deliver this information.
Although e.g. light detection and ranging (LIDAR) scans are used to capture 3D set information, frame by frame geometric information for dynamic scenes including moving camera, actors and props are not being recorded. Stereo camera systems deliver additional data for depth map generation but accuracy is limited.
This work suggests a method of capturing light field data within a regular live action shoot. We compute geometric data for post-production use cases such as relighting, depth-based compositing, 3D integration, virtual camera, digital focus as well as virtual backlots. Thus, the 2D high quality life action plate shot by the Director of Photography (DoP) comes along with image data from the light field array that can be further processed and utilized.
As a proof-of-concept we produced a fictitious commercial using a main camera and a multi-camera array fitted into an S3D mirror rig. For this reference movie, the image of the center array cam has been selected for further image manipulation. The quality of the depth information is evaluated on simulated data and live action footage. Our experiments show that quality of depth maps depends on array geometry and scene setup.

Supplementary Material

MP4 File (a3-ziegler.mp4)

References

[1]
Frick, A., Bartczack, B., Koch, R. 2010, 3D-TV LDV content generation with a hybrid ToF-multicamera RIG. 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), (Finland, Tampere, June 7-9 2010), DOI= 10.1109/3DTV.2010.5506604
[2]
Gandhi, V., Cech, J., Horaud, R., 2012, High resolution Depth Maps based on TOF-Stereo Fusion, International Conference on Robotics and Automation,(USA, Minnesota, Saint Paul, May 14-18 2012)
[3]
Grees, J. 2014.{digital} Visual Effects and Compositing. New Riders, San Francisco, CA
[4]
Hach, T., Steurer, J., A Novel RGB-Z Camera for High-Quality Motion Picture Applications, 2013, Proceedings of the 10th European Conference on Visual Media Production (CVMP 2013), ACM, New York, NY, USA, Article 4, 10 pages. DOI= 10.1145/2534008.2534020
[5]
Hartley, R. I., Zisserman, A. 2004. Multiple View Geometry in Computer Vision (2nd. ed.). Cambridge University Press, Cambridge.
[6]
Hirschmuller, H., Accurate and efficient stereo processing by semi-global matching and mutual information, Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 2, no., pp. 807, 814 vol. 2, 20-25 June 2005.
[7]
Ho, J., Kang, Y. S., 2010, Multi-View depth generation using multi-depth camera system, International Conference on 3D-Systems and Applications (3DSA), (Japan, Tokyo, May 19-21, 2010)
[8]
Jarabo, A., Masia, b., Gutierrez, Diego, 2011. Efficient Propagation of Light Field Edits. Proceedings of SIACG. Faro, Portugal: 75--80.
[9]
Kim, C., Zimmer, H., Pritch Y., Sorkine-Hornung, A., Gross M., 2013. Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph. 32, 4, Article 73 (July 2013), 12 pages. DOI=10.1145/2461912.2461926
[10]
Lee, E., Ho., Y. 2011, Generation of high-quality depth maps using hybrid camera system for 3-D video. Journal of Visual Communication and Image Representation.
[11]
Park, J., Hyeongwoo, K., Yu-Wing, T., Brown, M. S., Inso K. 2011, High quality depth map upsampling for 3D-TOF cameras. International Conference on Computer Vision (ICCV), (Barcelona, Spain, October 6-13 2011), pages 1623--1630, DOI=10.1109/ICCV.2011.6126423
[12]
Phong, T. B., Illumination for computer generated pictures. Commun. ACM 18, 6 (June 1975), 311--317. DOI=10.1145/360825.360839
[13]
Porter, T., Duff, T., 1984. Compositing Digital Images. SIGGRAPH Comput. Graph. 18.3: 253--259.
[14]
Saam, J., 2010. Creative relighting in compositing based on z depth, National center of computer animation Bournemouth University
[15]
Schnyder L., Lang, M., Wang, O., Smolic, A. 2012. Depth image based compositing for stereo 3D. 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), (Zurich, Switzerland, October 12-17, 2012). IEEE. DOI=10.1109/3DTV.2012.6365451
[16]
Scharstein D., Hirschmüller H., Kitajima Y., Krathwohl G., Nesic N., Wang X., Westling P., 2014, High-resolution stereo datasets with subpixel-accurate ground truth. German Conference on Pattern Recognition (GCPR 2014), Münster, Germany, September 2014.
[17]
Seitz, S. M., Curless, B., Diebel, J.; Scharstein, D., Szeliski, R., 2006, A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, no., pp. 519, 528, 17-22 June 2006, DOI = 10.1109/CVPR.2006.19
[18]
Seitz, S. M., Kutulakos K. N., 1998. Plenoptic Image Editing. Proc. ICCV 98. N.p.
[19]
Spears, D., 2010. 3D Compositing. The VES Handbook of Visual Effects. Eds. Jeffrey Okun and Susan Zwerman. Elsevier/Focal Press, Oxford. 681--690.
[20]
Spielmann, S., Helzle, V., Nair, R., 2013, On-set depth capturing for VFX productions using time of flight, ACM SIGGRAPH 2013 Talks (SIGGRAPH 2013), ACM, New York, NY, USA, Article 13, 1 page.
[21]
Tomasi, C.; Manduchi, R., 1998, Bilateral filtering for gray and color images, Computer Vision, Sixth International Conference on, pp. 839, 846, 4-7 Jan 1998
[22]
Um, G. M., Kim T., Bang, G., Hur, N., Lee, E. K., Jung, J. I., Kang, Y. S. Lee, G. Y., Ho, Y. S., 2011, Multi-View 3D Video acquisition using hybrid cameras with beam splitter, 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), (Turkey, Antalya, May 16-18 2011), IEEE.
[23]
Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., Debevec, P., 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans. Graph. 24, 3 (July 2005), 756--764. DOI=10.1145/1073204.1073258
[24]
Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M., 2005, High Performance Imaging Using Large Camera Arrays, ACM Transactions on Graphics, vol. 24, no. 3, pages 765--776, ACM, New York, NY, USA
[25]
Xing, M., Xun, S., Mingcai Z., Shaohui J., Haitao, W., Xiaopeng Z., 2011, On building an accurate stereo matching system on graphics hardware, Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, vol., no., pp. 467, 474, 6-13 Nov. 2011.
[26]
Zilly, F., Gutberlet, N. M., Riechert, C., Tanger, R., Kauff, P., 2012, Depth based content creation targeting stereoscopic and auto-stereoscopic displays, The Best of IET and IBC, The Institution of Engineering and Technology, vol. 4, page 8--14.
[27]
Zilly, F., Ziegler, M., Keinert, J., Schoeberl, M., Foessel, S., 2014, Computational imaging for stop-motion animated video productions, IET Conference Proceedings, (Amsterdam, Netherlands, September 11-15), page 12.3--12-3, DOI= 10.1049/ib.2014.003
[28]
Zilly, F., Riechert, C., Müller, M., Kauff, P., 2012, Multi-Camera Rectification using Linearized Trifocal Tensor, International Conference on Pattern Recognition (CVPR), (Tsukuba, Japan, November 11-15), page 2727--2731

Cited By

View all
  • (2022)Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.01798(18511-18520)Online publication date: Jun-2022
  • (2019)The Potential of Light Fields in Media ProductionsSIGGRAPH Asia 2019 Technical Briefs10.1145/3355088.3365158(71-74)Online publication date: 17-Nov-2019
  • (2018)3D Visual Content Datasets3D Visual Content Creation, Coding and Delivery10.1007/978-3-319-77842-6_11(299-325)Online publication date: 29-Jul-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
CVMP '15: Proceedings of the 12th European Conference on Visual Media Production
November 2015
121 pages
ISBN:9781450335607
DOI:10.1145/2824840
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

  • BMVA: British Machine Vision Association and Society for Pattern Recognition
  • Google Inc.
  • NVIDIA
  • CDE: Centre for Digital Entertainment
  • YouTube: YouTube
  • The Foundry: The Foundry Visionmongers Ltd.
  • Autodesk: Autodesk

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 November 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. computer vision
  2. lightfield
  3. visual effects

Qualifiers

  • Research-article

Conference

CVMP 2015
Sponsor:
  • BMVA
  • CDE
  • YouTube
  • The Foundry
  • Autodesk

Acceptance Rates

Overall Acceptance Rate 40 of 67 submissions, 60%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)22
  • Downloads (Last 6 weeks)4
Reflects downloads up to 08 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.01798(18511-18520)Online publication date: Jun-2022
  • (2019)The Potential of Light Fields in Media ProductionsSIGGRAPH Asia 2019 Technical Briefs10.1145/3355088.3365158(71-74)Online publication date: 17-Nov-2019
  • (2018)3D Visual Content Datasets3D Visual Content Creation, Coding and Delivery10.1007/978-3-319-77842-6_11(299-325)Online publication date: 29-Jul-2018
  • (2017)The Light Field Attachment: Turning a DSLR into a Light Field Camera Using a Low Budget Camera RingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2016.262874323:10(2357-2364)Online publication date: 1-Oct-2017
  • (2017)Characterization and selection of light field content for perceptual assessment2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX.2017.7965635(1-6)Online publication date: May-2017
  • (2017)Modeling depth uncertainty of desynchronized multi-camera systems2017 International Conference on 3D Immersion (IC3D)10.1109/IC3D.2017.8251891(1-6)Online publication date: Dec-2017
  • (2016)Computational Imaging for Stop-Motion Animated Video ProductionsSMPTE Motion Imaging Journal10.5594/j18663125:1(42-47)Online publication date: Jan-2016
  • (2016)Efficient Multi-image Correspondences for On-line Light Field Video ProcessingComputer Graphics Forum10.5555/3151666.315170735:7(401-410)Online publication date: 1-Oct-2016

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media