Skip to main content
SpringerLink
Log in

Fast Approximate Light Field Volume Rendering: Using Volume Data to Improve Light Field Synthesis via Convolutional Neural Networks

  • Conference paper
  • First Online:
Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1182))

  • 736 Accesses

Abstract

Volume visualization pipelines have the potential to be improved by the use of light field display technology, allowing enhanced perceptual qualities. However, these displays will require a significant increase in pixels to be rendered at interactive rates. Volume rendering makes use of ray-tracing techniques, which makes this resolution increase challenging for modest hardware. We demonstrate in this work an approach to synthesize the majority of the viewpoints in the light field using a small set of rendered viewpoints via a convolutional neural network. We show that synthesis performance can be further improved by allowing the network access to the volume data itself. To perform this efficiently, we propose a range of approaches and evaluate them against two datasets collected for this task. These approaches all improve synthesis performance and avoid the use of expensive 3D convolutional operations. With this approach, we improve light field volume rendering times by a factor of 8 for our test case.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Agus, M., et al.: An interactive 3D medical visualization system based on a light field display. Vis. Comput. 25(9), 883–893 (2009). https://doi.org/10.1007/s00371-009-0311-y

    Article  Google Scholar 

  2. Agus, M., Gobbetti, E., Guitián, J.A.I., Marton, F., Pintore, G.: GPU accelerated direct volume rendering on an interactive light field display. Comput. Graph. Forum 27(2), 231–240 (2008). https://doi.org/10.1111/j.1467-8659.2008.01120.x

    Article  Google Scholar 

  3. Bilen, H., Fernando, B., Gavves, E., Vedaldi, A., Gould, S.: Dynamic image networks for action recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3034–3042 (2016). https://doi.org/10.1109/CVPR.2016.331

  4. Birklbauer, C., Bimber, O.: Light-field supported fast volume rendering. In: ACM SIGGRAPH 2012 Posters on - SIGGRAPH 2012, p. 1. ACM Press, Los Angeles, California (2012). https://doi.org/10.1145/2342896.2343040

  5. Bruton, S., Ganter, D., Manzke, M.: Synthesising light field volumetric visualizations in real-time using a compressed volume representation. In: Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP. pp. 96–105. SciTePress (2019). https://doi.org/10.5220/0007407200960105

  6. Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. arXiv:1512.03012 [cs] (2015)

  7. Drebin, R.A., Carpenter, L., Hanrahan, P.: Volume rendering. In: Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques, pp. 65–74. SIGGRAPH 1988, ACM, New York, NY, USA (1988). https://doi.org/10.1145/54852.378484

  8. Engelmann, F., Kontogianni, T., Hermans, A., Leibe, B.: Exploring spatial context for 3D semantic segmentation of point clouds. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). pp. 716–724 (2017). https://doi.org/10.1109/ICCVW.2017.90

  9. Favalora, G.E.: Volumetric 3D displays and application infrastructure. Computer 38(8), 37–44 (2005). https://doi.org/10.1109/MC.2005.276

    Article  Google Scholar 

  10. Fernando, B., Gavves, E.M., Oramas, J., Ghodrati, A., Tuytelaars, T.: Rank pooling for action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 773–787 (2017). https://doi.org/10.1109/TPAMI.2016.2558148

    Article  Google Scholar 

  11. Fishman, E.K., Ney, D.R., Heath, D.G., Corl, F.M., Horton, K.M., Johnson, P.T.: Volume rendering versus maximum intensity projection in CT angiography: what works best, when, and why. Radiographics: A Review Publication of the Radiological Society of North America, Inc 26(3), 905–922 (2006). https://doi.org/10.1148/rg.263055186

    Article  Google Scholar 

  12. Hadwiger, M., Kratz, A., Sigg, C., Bühler, K.: GPU-accelerated deep shadow maps for direct volume rendering. In: Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, pp. 49–52. GH 2006, ACM, New York, NY, USA (2006). https://doi.org/10.1145/1283900.1283908

  13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  14. Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. 35(6), 193:1–193:10 (2016). https://doi.org/10.1145/2980179.2980251

    Article  Google Scholar 

  15. Kühnapfel, U., Çakmak, H.K., Maaß, H.: Endoscopic surgery training using virtual reality and deformable tissue simulation. Comput. Graph. 24(5), 671–682 (2000). https://doi.org/10.1016/S0097-8493(00)00070-4

    Article  Google Scholar 

  16. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 [cs] (2014)

  17. Klokov, R., Lempitsky, V.: Escape from cells: deep kd-networks for the recognition of 3D point cloud models. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 863–872 (2017). https://doi.org/10.1109/ICCV.2017.99

  18. Kniss, J., Kindlmann, G., Hansen, C.: Multidimensional transfer functions for interactive volume rendering. IEEE Trans. Vis. Comput. Graph. 8(3), 270–285 (2002). https://doi.org/10.1109/TVCG.2002.1021579

    Article  Google Scholar 

  19. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., New York (2012)

    Google Scholar 

  20. Lacroute, P., Levoy, M.: Fast volume rendering using a shear-warp factorization of the viewing transformation. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, pp. 451–458. SIGGRAPH 1994, ACM, New York, NY, USA (1994). https://doi.org/10.1145/192161.192283

  21. Lanman, D., Luebke, D.: Near-eye light field displays. In: ACM SIGGRAPH 2013 Emerging Technologies, p. 11:1. SIGGRAPH 2013, ACM, New York, NY, USA (2013). https://doi.org/10.1145/2503368.2503379

  22. Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 31–42. SIGGRAPH 1996, ACM, New York, NY, USA (1996). https://doi.org/10.1145/237170.237199

  23. Li, Y., Pirk, S., Su, H., Qi, C.R., Guibas, L.J.: FPNN: field probing neural networks for 3D data. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 307–315. Curran Associates, Inc., New York (2016)

    Google Scholar 

  24. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv:1312.4400 [cs] (2013)

  25. Liu, T.Y.: Learning to rank for information retrieval. Found. Trends Inf. Retr. 3(3), 225–331 (2009). https://doi.org/10.1561/1500000016

    Article  Google Scholar 

  26. Liu, Z., Yeh, R.A., Tang, X., Liu, Y., Agarwala, A.: Video frame synthesis using deep voxel flow. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4473–4481 (2017). https://doi.org/10.1109/ICCV.2017.478

  27. Ljung, P., Krüger, J., Groller, E., Hadwiger, M., Hansen, C.D., Ynnerman, A.: State of the art in transfer functions for direct volume rendering. Comput. Graph. Forum 35(3), 669–691 (2016). https://doi.org/10.1111/cgf.12934

    Article  Google Scholar 

  28. Mora, B., Maciejewski, R., Chen, M., Ebert, D.S.: Visualization and computer graphics on isotropically emissive volumetric displays. IEEE Trans. Vis. Comput. Graph. 15(2), 221–234 (2009). https://doi.org/10.1109/TVCG.2008.99

    Article  Google Scholar 

  29. Mueller, K., Yagel, R.: Fast perspective volume rendering with splatting by utilizing a ray-driven approach. In: Proceedings of Seventh Annual IEEE Visualization 1996, pp. 65–72 (1996). https://doi.org/10.1109/VISUAL.1996.567608

  30. Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive separable convolution. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 261–270 (2017). https://doi.org/10.1109/ICCV.2017.37

  31. Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C.: Transformation-grounded image generation network for novel 3D view synthesis. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 702–711 (2017). https://doi.org/10.1109/CVPR.2017.82

  32. Philips, S., Hlawitschka, M., Scheuermann, G.: Slice-based visualization of brain fiber bundles - a lic-based approach. In: Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, pp. 281–288. SciTePress (2018). https://doi.org/10.5220/0006619402810288

  33. Qi, C.R., Su, H., Kaichun, M., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85 (2017). https://doi.org/10.1109/CVPR.2017.16

  34. Qi, C.R., Su, H., Nießner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view CNNs for object classification on 3D data. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5648–5656 (2016). https://doi.org/10.1109/CVPR.2016.609

  35. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 5099–5108. Curran Associates, Inc., New York (2017)

    Google Scholar 

  36. Riegler, G., Ulusoy, A.O., Geiger, A.: OctNet: Learning deep 3D representations at high resolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6620–6629 (2017). https://doi.org/10.1109/CVPR.2017.701

  37. Salama, C.R.: GPU-based monte-carlo volume raycasting. In: 15th Pacific Conference on Computer Graphics and Applications (PG 2007). pp. 411–414 (2007). https://doi.org/10.1109/PG.2007.27

  38. Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004). https://doi.org/10.1023/B:STCO.0000035301.49549.88

    Article  MathSciNet  Google Scholar 

  39. Srinivasan, P.P., Wang, T., Sreelal, A., Ramamoorthi, R., Ng, R.: Learning to synthesize a 4D RGBD light field from a single image. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2262–2270 (2017). https://doi.org/10.1109/ICCV.2017.246

  40. Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 945–953. ICCV 2015, IEEE Computer Society, Washington, DC, USA (2015). https://doi.org/10.1109/ICCV.2015.114

  41. Sunden, E., et al.: Inviwo - an extensible, multi-purpose visualization framework. In: 2015 IEEE Scientific Visualization Conference (SciVis), pp. 163–164 (2015). https://doi.org/10.1109/SciVis.2015.7429514

  42. Tuzel, O., Liu, M.-Y., Taguchi, Y., Raghunathan, A.: Learning to rank 3D features. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 520–535. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_34

    Chapter  Google Scholar 

  43. Wang, P., Li, W., Gao, Z., Zhang, Y., Tang, C., Ogunbona, P.: Scene flow to action map: a new representation for RGB-D based action recognition with convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 416–425 (2017). https://doi.org/10.1109/CVPR.2017.52

  44. Wang, P.S., Liu, Y., Guo, Y.X., Sun, C.Y., Tong, X.: O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph. 36(4), 1–11 (2017). https://doi.org/10.1145/3072959.3073608

    Article  Google Scholar 

  45. Wetzstein, G., Lanman, D., Hirsch, M., Raskar, R.: Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. ACM Trans. Graph. 31(4), 80:1–80:11 (2012). https://doi.org/10.1145/2185520.2185576

    Article  Google Scholar 

  46. Xie, J., Dai, G., Zhu, F., Wong, E.K., Fang, Y.: Deepshape: deep-learned shape descriptor for 3D shape retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1335–1345 (2017). https://doi.org/10.1109/TPAMI.2016.2596722

    Article  Google Scholar 

  47. Zhang, Y., Dong, Z., Ma, K.: Real-time volume rendering in dynamic lighting environments using precomputed photon mapping. IEEE Trans. Vis. Comput. Graph. 19(8), 1317–1330 (2013). https://doi.org/10.1109/TVCG.2013.17

    Article  Google Scholar 

  48. Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. 37(4), 65:1–65:12 (2018). https://doi.org/10.1145/3197517.3201323

    Article  Google Scholar 

  49. Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_18

    Chapter  Google Scholar 

Download references

Acknowledgements

The authors would also like to thank the anonymous referees for their valuable comments and helpful suggestions. This research has been conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 13/IA/1895.

Author information

Authors and Affiliations

  1. School of Computer Science and Statistics, Trinity College Dublin, University of Dublin, Dublin, Ireland

    Seán Bruton, David Ganter & Michael Manzke

Authors
  1. Seán Bruton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Ganter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Manzke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seán Bruton .

Editor information

Editors and Affiliations

  1. University of Lisbon, Lisbon, Portugal

    Ana Paula Cláudio

  2. University of Rennes 1, Rennes, France

    Kadi Bouatouch

  3. University of Genoa, Genoa, Italy

    Manuela Chessa

  4. Mines ParisTech, Paris, France

    Alexis Paljic

  5. Linnaeus University, Växjö, Sweden

    Andreas Kerren

  6. French Civil Aviation University (ENAC), Toulouse, France

    Christophe Hurter

  7. University Jean Monnet, Saint-Etienne, France

    Alain Tremeau

  8. University of Catania, Catania, Italy

    Giovanni Maria Farinella

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bruton, S., Ganter, D., Manzke, M. (2020). Fast Approximate Light Field Volume Rendering: Using Volume Data to Improve Light Field Synthesis via Convolutional Neural Networks. In: Cláudio, A., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2019. Communications in Computer and Information Science, vol 1182. Springer, Cham. https://doi.org/10.1007/978-3-030-41590-7_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-41590-7_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-41589-1

  • Online ISBN: 978-3-030-41590-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation