skip to main content
10.1145/3084363.3085021acmconferencesArticle/Chapter ViewAbstractPublication PagessiggraphConference Proceedingsconference-collections
invited-talk

Large scale VFX pipelines

Published: 30 July 2017 Publication History

Abstract

To ensure peak utilization of hardware resources, as well as handle the increasingly dynamic demands placed on its render farm infrastructure, WETA Digital developed custom queuing, scheduling, job description and submission systems - which work in concert to maximize the available cores across a large range of non-uniform task types.
The render farm is one of the most important, high traffic components of a modern VFX pipeline. Beyond the hardware itself a render farm requires careful management and maintenance to ensure it is operating at peak efficiency. In WETAs case this hardware consists of a mix of over 80,000 CPU cores and a number of GPU resources, and as this has grown it has introduced many interesting scalability challenges.
In this talk we aim to present our end-to-end solutions in the render farm space, from the structure of the resource and the inherent problems introduced at this scale, through the development of Plow - our management, queuing and monitoring software. Finally we will detail the deployment process and production benefits realized. Within each section we intend to present the scalability issues encountered, and detail our strategy, process and results in solving these problems. The ever increasing complexity and computational demands of modern VFX drives WETAs need to innovate in all areas, from surfacing, rendering and simulation but also to core pipeline infrastructure.

Supplementary Material

MP4 File (talks-0002.mp4)

Cited By

View all
  • (2025)Intelligent Cinematography: a review of AI research for cinematographic productionArtificial Intelligence Review10.1007/s10462-024-11089-358:4Online publication date: 25-Jan-2025
  • (2023)The Evolution of the Virtual Production Studio as a Game Changer in FilmmakingCreating Digitally10.1007/978-3-031-31360-8_14(403-429)Online publication date: 3-Dec-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGGRAPH '17: ACM SIGGRAPH 2017 Talks
July 2017
158 pages
ISBN:9781450350082
DOI:10.1145/3084363
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2017

Check for updates

Author Tags

  1. distributed computing
  2. pipeline
  3. queuing
  4. scheduling

Qualifiers

  • Invited-talk

Conference

SIGGRAPH '17
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)13
  • Downloads (Last 6 weeks)5
Reflects downloads up to 27 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Intelligent Cinematography: a review of AI research for cinematographic productionArtificial Intelligence Review10.1007/s10462-024-11089-358:4Online publication date: 25-Jan-2025
  • (2023)The Evolution of the Virtual Production Studio as a Game Changer in FilmmakingCreating Digitally10.1007/978-3-031-31360-8_14(403-429)Online publication date: 3-Dec-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media