research-article

How much cloud is too much cloud?: What we wish we knew before we started

Authors:

Sam Bui,

Graeme DemmocksAuthors Info & Claims

DigiPro '23: Proceedings of the 2023 Digital Production Symposium

Article No.: 6, Pages 1 - 3

https://doi.org/10.1145/3603521.3604299

Published: 05 August 2023 Publication History

Get Access

Abstract

Avatar: The Way of Water, presented Wētā FX with the challenge of delivering visuals of an unprecedented level of complexity and quality. To support the filmmaker's creative vision, Wētā FX sought to ensure it had burst compute capacity available through scalable cloud resource.

In this talk we aim to present our approach to this problem in the cloud rendering space. We will provide an overview of the deployment of a large-scale hybrid cloud render-farm, diving into areas we found challenging and of particular interest to the wider VFX community. Namely the deployment of a custom-built NFS re-export cache tweaked to deal with high latency, our approach to maintaining performant storage in the cloud, our solution to scheduling a large-scale spot fleet across 3 AWS regions and various challenges encountered when rendering at scale in the cloud. Within each section we intend to present scalability issues encountered, detail various challenges and our approaches to overcome them.

References

[1]

Daire Byrne, DNEG. 2020. linux-nfs and linux-cachefs mailing list. Retrieved from https://lore.kernel.org/linux-nfs/[email protected]/T/

Google Scholar

[2]

Github, 2022. Google Cloud Platform knfsd project. Retrieved from https://github.com/GoogleCloudPlatform/knfsd-cache-utils

Google Scholar

Index Terms

How much cloud is too much cloud?: What we wish we knew before we started
1. Networks
  1. Network services
    1. Cloud computing

Recommendations

Evaluating and Optimizing Indexing Schemes for a Cloud-Based Elastic Key-Value Store
CCGRID '11: Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Cloud computing has emerged to provide virtual, pay-as-you-go computing and storage services over the Internet, where the usage cost directly depends on consumption. One compelling feature in Clouds is elasticity, where a user can demand, and be ...
DevOps patterns to scale web applications using cloud services
SPLASH '13: Proceedings of the 2013 companion publication for conference on Systems, programming, & applications: software for humanity

Scaling a web applications can be easy for simple CRUD software running when you use Platform as a Service Clouds (PaaS). But if you need to deploy a complex software, with many components and a lot users, you will need have a mix of cloud services in ...
Is your cloud elastic enough?: performance modelling the elasticity of infrastructure as a service (IaaS) cloud applications
ICPE '12: Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering

Elasticity, the ability to rapidly scale resources up and down on demand, is an essential feature of public cloud platforms. However, it is difficult to understand the elasticity requirements of a given application and workload, and if the elasticity ...

Comments

Information & Contributors

Information

Published In

DigiPro '23: Proceedings of the 2023 Digital Production Symposium

August 2023

84 pages

ISBN:9798400702389

DOI:10.1145/3603521

Editors:
Valérie Bernard
Animal Logic
,
Kelsey Hurley
Walt Disney Animation Studios
,
Cyrus Wilson,
Per Christensen
Pixar Animation Studios

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

DigiPro '23

Sponsor:

SIGGRAPH

DigiPro '23: The Digital Production Symposium

August 5, 2023

CA, Los Angeles, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
209
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Index Terms

Recommendations

Evaluating and Optimizing Indexing Schemes for a Cloud-Based Elastic Key-Value Store

DevOps patterns to scale web applications using cloud services

Is your cloud elastic enough?: performance modelling the elasticity of infrastructure as a service (IaaS) cloud applications

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations