Skip to main content
Springer Nature Link
Log in

UniTorch - Integrating Neural Rendering into Unity

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14361))

Included in the following conference series:

  • 1028 Accesses

Abstract

Neural rendering techniques have gained significant attention in recent years for their ability to generate highly realistic and immersive visual content. This paper discusses the current state of game engines regarding their ability to integrate neural modules within their pipelines. We exemplarily chose the popular game engine Unity and the deep learning library LibTorch. As we found a severe gap between commonly used auto-diff, deployment and rendering frameworks regarding interoperability and performance, we designed UniTorch, a plug-in that allows native access from Unity to Torch. We explore the practical integration of neural rendering methods by faithfully reimplementing and extending state-of-the-art methods. We provide detailed implementation guidelines and use it as means to reveal the mentioned gaps through extensive benchmarking experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Game On: Game-Engine Technology Expands Filmmaking Horizons - The American Society of Cinematographers (en-US). https://tinyurl.com/2txhdvac

  2. NeuralNetworkInference: Unreal Engine Doc. https://tinyurl.com/aa2ryfej

  3. Official - Introducing Unity Muse and Unity Sentis, AI-Powered Creativity. https://tinyurl.com/35c8eev8

  4. ONNX. https://onnx.ai/

  5. Torch—PyTorch 2.0 documentation. https://pytorch.org/docs/stable/torch.html

  6. Unreal Engine 5.2 Release Notes. https://tinyurl.com/yx5e6ba3

  7. Unity Barracuda. Unity Technologies (2021)

    Google Scholar 

  8. PyTorch Realtime Style Transfer Model in Unreal Engine 5 with ONNX Runtime (2022)

    Google Scholar 

  9. Neural Network Engine (NNE). Course (2023). https://tinyurl.com/fejs7pem

  10. Unreal Engine. Wikipedia (2023)

    Google Scholar 

  11. Božič, A., Gladkov, D., Doukakis, L., Lassner, C.: Neural assets: volumetric object capture and rendering for interactive environments (2022)

    Google Scholar 

  12. Chen, Z., Funkhouser, T., Hedman, P., Tagliasacchi, A.: MobileNeRF: exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16569–16578 (2023)

    Google Scholar 

  13. Frames, W.: Bringing Deep Learning to Unreal Engine 5—Pt. 2 (2022)

    Google Scholar 

  14. Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: radiance fields without neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5501–5510 (2022)

    Google Scholar 

  15. Gao, K., Gao, Y., He, H., Lu, D., Xu, L., Li, J.: NeRF: neural radiance field in 3D vision, a comprehensive review (2022)

    Google Scholar 

  16. Guttenberg, N.: Neural networks in unity using native libraries. https://www.goodai.com/neural-networks-in-unity-using-native-libraries/

  17. Hedman, P., Srinivasan, P.P., Mildenhall, B., Barron, J.T., Debevec, P.: Baking neural radiance fields for real-time view synthesis. In: IEEE/CVF International Conference on Computer Vision (ICCV), Virtual, pp. 5875–5884 (2021)

    Google Scholar 

  18. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV (2017)

    Google Scholar 

  19. Kipp, J.: MobileNeRF in Unity (2022). https://t.co/SslORxUbFJ

  20. Kipp, J.: SNeRG Unity Viewer (2023). https://github.com/julienkay/SNeRG-Unity-Viewer

  21. Lavik, M.: UnityVolumeRendering (2023). https://tinyurl.com/bdf8vhp3

  22. Li, K., et al.: Bringing instant neural graphics primitives to immersive virtual reality. In: 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Shanghai, China, pp. 739–740. IEEE (2023)

    Google Scholar 

  23. Marschner, S., Shirley, P.: Fundamentals of Computer Graphics. CRC Press, Taylor & Francis Group (2022)

    Google Scholar 

  24. Marshall, C.S.: Practical machine learning for rendering: from research to deployment. In: ACM SIGGRAPH 2021 Courses, Virtual Event, USA, pp. 1–239. ACM (2021)

    Google Scholar 

  25. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  26. Mills, C.: Create a libtorch plugin for unity. https://christianjmills.com/posts/fastai-libtorch-unity-tutorial/part-1/

  27. Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41(4), 1–15 (2022)

    Article  Google Scholar 

  28. Pokhrel, C., Khatiwada, A.: Deep Q-learning for intelligent non-playable characters in combat games (2023)

    Google Scholar 

  29. Qiu, W., Yuille, A.: UnrealCV: connecting computer vision to unreal engine. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 909–916. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49409-8_75

    Chapter  Google Scholar 

  30. Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: ground truth from computer games. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 102–118. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_7

    Chapter  Google Scholar 

  31. Saint-Denis, A., Vanhoey, K., Deliot, T.: Multi-stylization of video-games in real-time guided by G-buffer information. In: High Performance Graphics 2019, Strasbourg, France (2019)

    Google Scholar 

  32. Szlęg, P., Barczyk, P., Maruszczak, B., Zieliñski, S., Szymañska, E.: Simulation environment for underwater vehicles testing and training in Unity3D. In: Petrovic, I., Menegatti, E., Marković, I. (eds.) IAS 2022. LNNS, vol. 577, pp. 844–853. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-22216-0_56

    Chapter  Google Scholar 

  33. Tewari, A., et al.: State of the art on neural rendering. In: Computer Graphics Forum, vol. 39, no. 2, pp. 701–727 (2020)

    Google Scholar 

  34. Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. 38(4), 1–12 (2019)

    Article  Google Scholar 

  35. Unity Technologies: Manual: Render pipelines. https://docs.unity3d.com/Manual/render-pipelines.html

  36. NeuralVFX: basic libtorch dll. https://github.com/NeuralVFX/basic-libtorch-dll

  37. Yuan, L.: A Brief History of Deep Learning Frameworks (2021). https://tinyurl.com/46zb9yfm

Download references

Author information

Authors and Affiliations

  1. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Laura Fink, Daniel Keitel & Marc Stamminger

  2. Fraunhofer IIS, Erlangen, Germany

    Laura Fink, Daniel Keitel & Joachim Keinert

Authors
  1. Laura Fink
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Keitel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marc Stamminger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joachim Keinert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Fink .

Editor information

Editors and Affiliations

  1. University of Nevada Reno, Reno, NV, USA

    George Bebis

  2. Google Research, Mountain View, CA, USA

    Golnaz Ghiasi

  3. New York University, New York, USA

    Yi Fang

  4. Ben-Gurion University, Be'er Sheva, Israel

    Andrei Sharf

  5. Microsoft Research, Beijing, China

    Yue Dong

  6. The University of Oklahoma, Norman, OK, USA

    Chris Weaver

  7. University of Maryland, Collage Park, MD, USA

    Zhicheng Leo

  8. University of Central Florida, Orlando, FL, USA

    Joseph J. LaViola Jr.

  9. InnerOptic Technology, Hillsborough, NC, USA

    Luv Kohli

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fink, L., Keitel, D., Stamminger, M., Keinert, J. (2023). UniTorch - Integrating Neural Rendering into Unity. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2023. Lecture Notes in Computer Science, vol 14361. Springer, Cham. https://doi.org/10.1007/978-3-031-47969-4_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47969-4_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47968-7

  • Online ISBN: 978-3-031-47969-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics