Skip to main content
SpringerLink
Log in

Deep generative smoke simulator: connecting simulated and real data

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We propose a novel generative adversarial architecture to generate realistic smoke sequences. Physically based smoke simulation methods are difficult to match with real-captured data since smoke is vulnerable to disturbance. In our work, we design a generator that takes into account the temporal movement of smoke as well as detailed structures. With the help of convolutional neural networks and long short-term memory-based autoencoder, our generator can predict the future frames using temporal information while preserving details. We use generative adversarial networks to train the model on both simulated and real-captured data and propose a combined loss function that reflects both the physical laws and the data distributions. We also demonstrate a multi-phase training strategy that significantly speeds up convergence and increases stability of training on real-captured data. To test our approach, we set up experiments to capture real smoke sequences and show that our method can achieve realistic visual effects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: a system for large-scale machine learning. OSDI 16, 265–283 (2016)

    Google Scholar 

  2. Angelidis, A., Neyret, F.: Simulation of smoke based on vortex filament primitives. In: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on computer animation, ACM, pp. 87–96 (2005)

  3. Angelidis, A., Neyret, F., Singh, K., Nowrouzezahrai, D.: A controllable, fast and stable basis for vortex based smoke simulation. In: Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on computer animation, Eurographics Association, pp. 25–32 (2006)

  4. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. (2017). arXiv:1701.07875

  5. Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)

    Article  Google Scholar 

  6. Brackbill, J., Ruppel, H.: Flip: a method for adaptively zoned, particle-in-cell calculations of fluid flows in two dimensions. J. Comput. Phys. 65(2), 314–343 (1986)

    Article  MathSciNet  Google Scholar 

  7. Brochu, T., Keeler, T., Bridson, R.: Linear-time smoke animation with vortex sheet meshes. In: Proceedings of the ACM SIGGRAPH/Eurographics symposium on computer animation, Eurographics Association, pp. 87–95 (2012)

  8. Browning, M., Barnes, C., Ritter, S., Finkelstein, A.: Stylized keyframe animation of fluid simulations. In: Proceedings of the workshop on non-photorealistic animation and rendering, ACM, pp. 63–70 (2014)

  9. Chu, M., Thuerey, N.: Data-driven synthesis of smoke flows with cnn-based feature descriptors. ACM Trans. Graph. 36(4), 69 (2017)

    Article  Google Scholar 

  10. Digital meals Smoke atmosphere. (2012). http://digitalmeals.blogspot.com/2012/03/smoke-atmosphere.html

  11. Farimani, A.B., Gomes, J., Pande, V.S.: Deep learning the physics of transport phenomena. (2017). arXiv:1709.02432

  12. Fedkiw, R., Stam, J., Jensen, H.W.: Visual simulation of smoke. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques, ACM, pp. 15–22 (2001)

  13. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014). http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf

  14. Gregson, J., Krimerman, M., Hullin, M.B., Heidrich, W.: Stochastic tomography and its applications in 3d imaging of mixing fluids. ACM Trans. Graph 31(4), 52–61 (2012)

    Article  Google Scholar 

  15. Gregson, J., Ihrke, I., Thuerey, N., Heidrich, W.: From capture to simulation: connecting forward and inverse problems in fluids. ACM Trans. Graph. 33(4), 139 (2014)

    Article  Google Scholar 

  16. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: Advances in Neural Information Processing Systems, vol. 30, pp. 5769–5779 (2017)

  17. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  18. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. (2017). arXiv Preprint

  19. Jeong, S., Solenthaler, B., Pollefeys, M., Gross, M., et al.: Data-driven fluid simulations using regression forests. ACM Trans. Graph. 34(6), 199 (2015)

    Google Scholar 

  20. Kalchbrenner, N., Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., Kavukcuoglu, K.: Video pixel networks. (2016). arXiv:1610.00527

  21. Kim, B., Liu, Y., Llamas, I., Rossignac, J.R.: Flowfixer: Using bfecc for fluid simulation. Technical report, Georgia Institute of Technology (2005)

  22. Kim, B., Azevedo, V.C., Thuerey, N., Kim, T., Gross, M., Solenthaler, B.: Deep fluids: a generative network for parameterized fluid simulations. (2018). arXiv:1806.02071

  23. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646–1654 (2016)

  24. Kim, T., Thürey, N., James, D., Gross, M.: Wavelet turbulence for fluid simulation. In: ACM Trans. Graph. (2008). https://doi.org/10.1145/1360612.1360649

  25. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. (2013). arXiv:1312.6114

  26. Koshizuka, S., Oka, Y.: Moving-particle semi-implicit method for fragmentation of incompressible fluid. Nucl. Sci. Eng. 123(3), 421–434 (1996)

    Article  Google Scholar 

  27. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in neural information processing systems, vol. 25, pp. 1097–1105 (2012)

  28. Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., Levine, S.: Stochastic adversarial video prediction. (2018). arXiv:1804.01523

  29. Li, X., Liu, L., Wu, W., Liu, X., Wu, E.: Dynamic bfecc characteristic mapping method for fluid simulations. Vis. Comput. 30(6–8), 787–796 (2014)

    Article  Google Scholar 

  30. Long, Z., Lu, Y., Ma, X., Dong, B.: Pde-net: learning pdes from data. (2017). arXiv:1710.09668

  31. Lotter, W., Kreiman, G., Cox, D.: Unsupervised learning of visual structure using predictive generative networks. (2015). arXiv:1511.06380

  32. Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising using separable 4d nonlocal spatiotemporal transforms. In: Image processing: algorithms and systems IX, International Society for Optics and Photonics, vol. 7870, p. 787003 (2015)

  33. Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms. IEEE Trans. Image Process. 21(9), 3952–3966 (2012). https://doi.org/10.1109/TIP.2012.2199324

  34. Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. (2015). arXiv:1511.05440. https://doi.org/10.1109/TIP.2012.2199324

  35. McAdams, A., Sifakis, E., Teran, J.: A parallel multigrid Poisson solver for fluids simulation on large grids. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics symposium on computer animation, Eurographics Association, pp. 65–74 (2010)

  36. Meng, Z., Weixin, S., Yinling, Q., Hanqiu, S., Jing, Q., Heng, P.A.: Vortex filaments in grids for scalable, fine smoke simulation. IEEE Comput. Graph. Appl. 35(6), 60–68 (2015)

    Article  Google Scholar 

  37. Mercier, O., Beauchemin, C., Thuerey, N., Kim, T., Nowrouzezahrai, D.: Surface turbulence for particle-based liquid simulations. ACM Trans. Graph. 34(6), 202 (2015)

    Article  Google Scholar 

  38. Mirza, M., Osindero, S.: Conditional generative adversarial nets. (2014). arXiv:1411.1784

  39. Okabe, M., Anjyor, K., Onai, R.: Creating fluid animation from a single image using video database. Comput. Graph. Forum. 30(7), 1973–1982 (2011). https://doi.org/10.1111/j.1467-8659.2011.02062.x

  40. Okabe, M., Dobashi, Y., Anjyo, K., Onai, R.: Fluid volume modeling from sparse multi-view images by appearance transfer. ACM Trans. Graph. 34(4), 93 (2015)

    Article  Google Scholar 

  41. Pfaff, T., Thuerey, N., Gross, M.: Lagrangian vortex sheets for animating fluids. ACM Trans. Graph. 31(4), 112 (2012)

    Article  Google Scholar 

  42. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. (2015). arXiv:1511.06434

  43. Ranzato, M., Szlam, A., Bruna, J., Mathieu, M., Collobert, R., Chopra, S.: Video (language) modeling: a baseline for generative models of natural videos. (2014). arXiv:1412.6604

  44. Rasmussen, N., Nguyen, D.Q., Geiger, W., Fedkiw, R: Smoke simulation for large scale phenomena. ACM Trans. Graph. 22(3), 703–707 (2003). https://doi.org/10.1145/882262.882335

  45. Saito, M., Matsumoto, E., Saito, S.: Temporal generative adversarial nets with singular value clipping. In: IEEE international conference on computer vision (ICCV), pp. 2830–2839 (2017)

  46. Selle, A., Rasmussen, N., Fedkiw, R.: A vortex particle method for smoke, water and explosions. ACM Trans. Graph. 24(3), 910–914 (2005). https://doi.org/10.1145/1073204.1073282

  47. Selle, A., Fedkiw, R., Kim, B., Liu, Y., Rossignac, J.: An unconditionally stable maccormack method. J. Sci. Comput. 35(2–3), 350–371 (2008)

    Article  MathSciNet  Google Scholar 

  48. Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.-C.: Convolutional lstm network: a machine learning approach for precipitation nowcasting. Adv. Neural Inf. Process. Syst. 28, 802–810 (2015)

    Google Scholar 

  49. Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R.: Learning from simulated and unsupervised images through adversarial training. In: The IEEE conference on computer vision and pattern recognition (CVPR), vol. 3, p. 6 (2017)

  50. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. (2014). arXiv:1409.1556

  51. Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using lstms. In: International conference on machine learning, pp. 843–852 (2015)

  52. Stam, J.: Stable fluids. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., pp. 121–128 (2015)

  53. Thuerey, N.: Interpolations of smoke and liquid simulations. ACM Trans. Graph. 36(1), 3 (2017)

    Article  Google Scholar 

  54. Thuong, H.: Video background HD—smoke HD—style proshow. (2014). https://www.youtube.com/watch?v=B6H34IccWks

  55. Tompson, J., Schlachter, K., Sprechmann, P., Perlin, K.: Accelerating Eulerian fluid simulation with convolutional networks. (2016). arXiv:1607.03597

  56. Um, K., Hu, X., Thuerey, N.: Liquid splash modeling with neural networks. Comput. Graph. Forum. 37(8), 171–182 (2018). https://doi.org/10.1111/cgf.13522

    Article  Google Scholar 

  57. Wang, C., Wang, C., Qin, H., Zhang, T.: Video-based fluid reconstruction and its coupling with sph simulation. Vis. Comput. 33(9), 1211–1224 (2017)

    Article  Google Scholar 

  58. Wen, J., Ma, H.: Real-time smoke simulation based on vorticity preserving lattice Boltzmann method. Vis.Comput. 35(9), 1279–1292 (2019)

  59. Wiewel, S., Becher, M., Thuerey, N.: Latent-space physics: Towards learning the temporal evolution of fluid flow. (2018). arXiv:1802.10123

  60. Xie, Y., Franz, E., Chu, M., Thuerey, N.: tempogan: a temporally coherent, volumetric gan for super-resolution fluid flow. (2018). arXiv:1801.09710

  61. Yoon, J.C., Kam, H.R., Hong, J.M., Kang, S.J., Kim, C.H.: Procedural synthesis using vortex particle method for fluid simulation. Comput. Graph. Forum. 28(7), 1853–1859 (2009). https://doi.org/10.1111/j.1467-8659.2009.01563.x

    Article  Google Scholar 

  62. Zhang, X., Bridson, R., Greif, C.: Restoring the missing vorticity in advection-projection fluid solvers. ACM Trans. Graph. 34(4), 52 (2015)

    Google Scholar 

  63. Zhu, Y., Bridson, R.: Animating sand as a fluid. ACM Trans. Graph. 24(3), 965–972 (2005)

    Article  Google Scholar 

Download references

Funding

This work was funded by National Key Basic Research Program of China (No. 2016YFB0100900) and National Natural Science Foundation of China (No. 61773231).

Author information

Authors and Affiliations

  1. Tsinghua University, Beijing, China

    Jinghuan Wen & Huimin Ma

  2. University of Science and Technology Beijing, Beijing, China

    Xiong Luo

Authors
  1. Jinghuan Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huimin Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huimin Ma.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 63522 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, J., Ma, H. & Luo, X. Deep generative smoke simulator: connecting simulated and real data. Vis Comput 36, 1385–1399 (2020). https://doi.org/10.1007/s00371-019-01738-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01738-y

Keywords

Search

Navigation