Skip to main content
SpringerLink
Log in

Highly parallel steered mixture-of-experts rendering at pixel-level for image and light field data

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

A novel image approximation framework called steered mixture-of-experts (SMoE) was recently presented. SMoE has multiple applications in coding, scale-conversion, and general processing of image modalities. In particular, it has strong potential for coding and streaming higher dimensional image modalities that are necessary to leverage full translational and rotational freedom (6 degrees-of-freedom) in virtual reality for camera captured images. In this paper, we analyze the rendering performance of SMoE for 2D images and 4D light fields. Two different GPU implementations that parallelize the SMoE regression step at pixel-level are presented, including experimental evaluations based on rendering performance and quality. In this paper it is shown that on appropriate hardware, an OpenCL implementation can achieve 85 fps and 22 fps for, respectively, 1080p and 4K renderings of large models with more than 100,000 of Gaussian kernels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ihrke, I., Restrepo, J., Mignard-Debise, L.: Principles of light field imaging: briefly revisiting 25 years of research. IEEE Signal Process. Mag. 33(5), 59–69 (2016). https://doi.org/10.1109/MSP.2016.2582220

    Article  Google Scholar 

  2. Adelson, E.H., Bergen, J.R.: The plenoptic function and the elements of early vision. In: Landy, M.S., Movshon, J.A. (eds.) Computational Models of Visual Processing, pp. 3–20. The MIT Press, Cambridge, MA, USA (1991)

    Google Scholar 

  3. Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques-SIGGRAPH ’96, pp. 31–42. ACM Press, New York (1996). https://doi.org/10.1145/237170.237199

  4. Wu, G., Masia, B., Jarabo, A., Zhang, Y., Wang, L., Dai, Q., Chai, T., Liu, Y.: Light field image processing: an overview. IEEE J. Sel. Top. Signal Process. 11(7), 926–954 (2017). https://doi.org/10.1109/JSTSP.2017.2747126

    Article  Google Scholar 

  5. Verhack, R., Sikora, T., Lange, L., Van Wallendael, G., Lambert, P.: A universal image coding approach using sparse mixture-of-experts regression. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 2142–2146. USA, Phoenix, AZ (2016). https://doi.org/10.1109/ICIP.2016.7532737

    Chapter  Google Scholar 

  6. Lange, L., Verhack, R., Sikora, T.: Video representation and coding using a sparse steered mixture-of-experts network. In: Picture Coding Symposium (PCS), pp. 1–5, Nuremberg, Germany (2016). https://doi.org/10.1109/PCS.2016.7906369

  7. Verhack, R., Sikora, T., Lange, L., Jongebloed, R., Van Wallendael, G., Lambert, P.: Steered mixture-of-experts for light field coding, depth estimation, and processing. In: Proceedings of the IEEE Conference on Multimedia and Expo (ICME), pp. 1183–1188, Hong Kong, China (2017). https://doi.org/10.1109/ICME.2017.8019442

  8. Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22(12), 1649–1668 (2012). https://doi.org/10.1109/TCSVT.2012.2221191

    Article  Google Scholar 

  9. Domanski, M., Stankiewicz, O., Wegner, K., Grajek, T.: Immersive visual media MPEG-I: 360 video, virtual navigation and beyond. In: 2017 International Conference on Systems, Signals and Image Processing (IWSSIP), pp. 1–9. IEEE, New York (2017). https://doi.org/10.1109/IWSSIP.2017.7965623

  10. Hinds, A.T., Doyen, D., Carballeira, P.: Toward the realization of six degrees-of-freedom with compressed light fields. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 1171–1176. IEEE, New York (2017). https://doi.org/10.1109/ICME.2017.8019543

  11. Stone, J.E., Gohara, D., Shi, G.: OpenCL: a parallel programming standard for heterogeneous computing systems. Comput. Sci. Eng. 12(3), 66–73 (2010). https://doi.org/10.1109/MCSE.2010.69

    Article  Google Scholar 

  12. Goossens, B., De Vylder, J., Philips, W.: Quasar—a new heterogeneous programming framework for image and video processing algorithms on CPU and GPU. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 2183–2185, Paris, France (2014). https://doi.org/10.1109/ICIP.2014.7025441

  13. Goossens, B.: Dataflow management, dynamic load balancing and concurrent processing for real-time embedded vision applications using quasar. Int. J. Circuit Theory Appl. 46(9), 1733–1755 (2018). https://doi.org/10.1002/cta.2494

    Article  Google Scholar 

  14. MATLAB: Version 8.6.0 (R2015b). The MathWorks Inc., Natick (2015)

  15. Smola, A., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004)

    Article  MathSciNet  Google Scholar 

  16. Broomhead, D.S., Lowe, D.: Radial basis functions, multi-variable functional interpolation and adaptive networks. Technical report, DTIC Document (1988)

  17. Altinigneli, M.C., Plant, C., Böhm, C.: Massively parallel expectation maximization using graphics processing units. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 838–846, ACM, New York (2013)

  18. Moon, T.: The expectation–maximization algorithm. IEEE Signal Process. Mag. 13(6), 47–60 (1996). https://doi.org/10.1109/79.543975

    Article  Google Scholar 

  19. Kumar, N.S.L.P., Satoor, S., Buck, I.: Fast parallel expectation maximization for Gaussian mixture models on GPUs using CUDA. In: 2009 11th IEEE International Conference on High Performance Computing and Communications, pp. 103–109. IEEE, New York (2009). https://doi.org/10.1109/HPCC.2009.45. http://ieeexplore.ieee.org/document/5166982/

  20. Verhack, R., Sikora, T., Lange, L., Van Wallendael, G., Lambert, P.: Steered mixture-of-experts for 4-D light field approximation, coding, and description. IEEE Trans. Multimedia (2018) (submitted)

  21. Yuksel, S.E., Wilson, J.N., Gader, P.D.: Twenty years of mixture of experts. IEEE Trans Neural Netw. Learn. Syst. 23(8), 1177–1193 (2012). https://doi.org/10.1109/TNNLS.2012.2200299

    Article  Google Scholar 

  22. Viola, I., Rerabek, M., Bruylants, T., Schelkens, P., Pereira, F., Ebrahimi, T.: Objective and subjective evaluation of light field image compression algorithms. In: 32nd Picture Coding Symposium (PCS), pp. 1–5, Nuremberg, Germany (2016)

  23. Rerabek, M., Ebrahimi, T.: New light field image dataset. In: 8th International Conference on Quality of Multimedia Experience (QoMEX). EPFL-CONF-218363, Lisbon, Portugal (2016). http://mmspg.epfl.ch/EPFL-light-field-image-dataset

  24. Tok, M., Jongebloed, R., Lange, L., Bochinski, E., Sikora, T.: An MSE approach for training and coding steered mixtures of experts. In: Picture Coding Symposium (PCS), pp. 273–277, San Francisco, CA, USA (2018). https://doi.org/10.1109/PCS.2018.8456250

  25. Bochinski, E., Jongebloed, R., Tok, M., Sikora, T.: Regularized gradient descent training of steered mixture of experts for sparse image representation. In: Accepted for Publication in IEEE International Conference on Image Processing (ICIP ’18), Athens, Greece (2018)

  26. Sung, H.: Gaussian Mixture Regression and Classification. Ph.D. thesis, Rice University (2004)

  27. Bugmann, G.: Normalized Gaussian radial basis function networks. Neurocomputing 20(1–3), 97–110 (1998). https://doi.org/10.1016/S0925-2312(98)00027-7

    Article  Google Scholar 

  28. Trefethen, L.N., Bau, D.: Numerical Linear Algebra. SIAM, Philadelphia (1997)

    Book  Google Scholar 

  29. Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes 3rd Edition: The Art of Scientific Computing. Cambridge University Press, Cambridge (2007)

    MATH  Google Scholar 

  30. Guennebaud, G., Jacob, B., et al.: Eigen v3. (2010). http://eigen.tuxfamily.org. Accessed 14 Dec 2018

  31. Wilt, N.: The CUDA Handbook: A Comprehensive Guide to GPU Programming, 2nd edn. Addison Wesley, Boston (2018)

    Google Scholar 

  32. Khronos OpenCL Working Group: The OpenCL Specification. Version 2.0. Revision 29. Technical report (2015)

  33. NVIDIA Corporation: CUDA occupancy calculator (2015). https://developer.nvidia.com/. Accessed 14 Dec 2018

  34. Volkov, V.: Better performance at lower occupancy. In: Proceedings of the GPU Technology Conference, GTC, vol. 10 (2010)

  35. Goossens, B.: Quasar—optimization guide (2017). https://quasar.ugent.be/files/doc/OptimizationGuide.html. Accessed 14 Dec 2018

  36. Vinkler, M., Havran, V.: Register efficient dynamic memory allocator for GPUs. Comput. Graph. Forum 34, 143–154 (2015)

    Article  Google Scholar 

  37. Altera, S.: For OpenCL Best Practices Guide. Programming Reference, Altera, May 2015 (2014)

  38. Takizawa, H., Hirasawa, S., Sugawara, M., Gelado, I., Kobayashi, H., Hwu, W.M.W.: Optimized data transfers based on the OpenCL event management mechanism. Sci. Program. 2015, 2 (2015). https://doi.org/10.1155/2015/576498

    Article  Google Scholar 

  39. Richter, T., Pinheiro, A., Schelkens, P., Skodras, A., Ebrahimi, T.: Image compression grand challenge at ICIP 2016. Technical report (2016)

Download references

Funding

The research activities described in this paper were funded by IDLab (Ghent University-imec), Flanders Innovation and Entrepreneurship (VLAIO), the Fund for Scientific Research Flanders (FWO Flanders), and the European Union.

Author information

Authors and Affiliations

  1. Department of Electronic and Information Systems, Ghent University-imec-IDLab, Technologiepark-Zwijnaarde 19, 9052, Ghent, Belgium

    Vasileios Avramelos, Ruben Verhack, Ignace Saenen, Glenn Van Wallendael & Peter Lambert

  2. Department of Telecommunications and Information Processing, Ghent University-imec-IPI, Sint-Pietersnieuwstraat 41B, 9000, Ghent, Belgium

    Bart Goossens

  3. Communications Systems Group, Technische Universität Berlin, Einsteinufer 17, 10587, Berlin, Germany

    Ruben Verhack

Authors
  1. Vasileios Avramelos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruben Verhack
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ignace Saenen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Glenn Van Wallendael
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bart Goossens
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peter Lambert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vasileios Avramelos.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Avramelos, V., Verhack, R., Saenen, I. et al. Highly parallel steered mixture-of-experts rendering at pixel-level for image and light field data. J Real-Time Image Proc 17, 931–947 (2020). https://doi.org/10.1007/s11554-018-0843-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-018-0843-3

Keywords

Search

Navigation