Skip to main content
SpringerLink
Log in

Rendering techniques for mixed reality

  • Special Issue
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In mixed reality (MR) design review, the aesthetics of a virtual prototype is assessed by integrating a virtual model into a real-world environment and inspecting the interaction between the model and the environment (lighting, shadows and reflections) from different points of view. The visualization of the virtual model has to be as realistically as possible to provide a solid basis for this assessment and interactive rendering speed is mandatory to allow the designer to examine the scene from arbitrary positions. In this article we present a real-time rendering engine specifically tailored to the needs of MR visualization. The renderer utilizes pre-computed radiance transfer to calculate dynamic soft-shadows, high dynamic range images and image-based lighting to capture incident real-world lighting, approximate bidirectional texture functions to render materials with self-shadowing, and frame post-processing filters (bloom filter and an adaptive tone mapping operator). The proposed combination of rendering techniques provides a trade-off between rendering quality and required computing resources which enables high quality rendering in mobile MR scenarios. The resulting image fidelity is superior to radiosity-based techniques because glossy materials and dynamic environment lighting with soft-shadows are supported. Ray tracing-based techniques provide higher quality images than the proposed system, but they require a cluster of computers to achieve interactive frame rates which prevents these techniques from being used in mobile MR (especially outdoor) scenarios. The renderer was developed in the European research project IMPROVE (FP6-IST-2-004785) and is currently extended in the MAXIMUS project (FP7-ICT-1-217039) where hybrid rendering techniques which fuse PRT and ray tracing are developed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. The screenshots were made using the VRED editor which we currently use for shader development and material application with OpenSG models.

References

  1. ARIS Project: European research project ARIS, augmented reality image synthesis through illumination reconstruction and its integration in interactive and shared mobile ar-systems for e-(motion)-commerce applications. http://aris-ist.intranet.gr (2004)

  2. Blinn, J.F.: Simulation of wrinkled surfaces. In: SIGGRAPH ’78: Proceedings of the 5th Annual Conference on Computer Graphics and Interactive Techniques, pp. 286–292. ACM Press, New York (1978). doi:http://doi.acm.org/10.1145/800248.507101

  3. Blinn, J.F., Newell, M.E.: Texture and reflection in computer generated images. In: SIGGRAPH ’76: Proceedings of the 3rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 266–266. ACM Press, New York (1976). doi:http://doi.acm.org/10.1145/563274.563322

  4. Caustic Graphics.: (2009). http://caustic.com/docs_faq.php

  5. Cook, R.L., Porter, T., Carpenter, L.: Distributed ray tracing. In: SIGGRAPH ’84: Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, pp. 137–145. ACM Press, New York (1984). doi:http://doi.acm.org/10.1145/800031.808590

  6. Dana, K.J., Nayar, S.K., Ginneken, B.V., Koenderink, J.J.: Reflectance and texture of real-world surfaces. In: CVPR ’97: Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR ’97), p. 151. IEEE Computer Society, Washington, DC (1997)

  7. Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: SIGGRAPH ’98: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, pp. 189–198. ACM Press, New York (1998). doi:http://doi.acm.org/10.1145/280814.280864

  8. Drettakis, G., Sillion, F.X.: Interactive update of global illumination using a line-space hierarchy. In: SIGGRAPH ’97: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 57–64. ACM Press, New York (1997). doi:http://doi.acm.org/10.1145/258734.258772

  9. Fiat Elasis.: (2009). http://www.elasis.it

  10. Franke, T., Jung, Y.: Precomputed radiance transfer for x3d based mixed reality applications. In: Web3D ’08: Proceedings of the 13th International Symposium on 3D Web Technology, pp. 7–10. ACM Press, New York (2008). doi:http://doi.acm.org/10.1145/1394209.1394213

  11. Goral, C.M., Torrance, K.E., Greenberg, D.P., Battaile, B.: Modeling the interaction of light between diffuse surfaces. In: SIGGRAPH ’84: Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, pp 213–222. ACM Press, New York (1984). doi:http://doi.acm.org/10.1145/800031.808601

  12. Hanrahan, P., Salzman, D., Aupperle, L.: A rapid hierarchical radiosity algorithm. In: SIGGRAPH ’91: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, pp. 197–206. ACM Press, New York (1991). doi:http://doi.acm.org/10.1145/122718.122740

  13. HDRShop: (2009). http://www.hdrshop.com

  14. Jensen, H.W.: Realistic image synthesis using photon mapping. A. K. Peters Ltd, Natick (2001)

  15. Kajiya, J.T.: The rendering equation. SIGGRAPH Comput. Graph. 20(4), 143–150 (1986). doi:http://doi.acm.org/10.1145/15886.15902

  16. Kautz, J.: Approximate bidirectional texture functions. In: Pharr, M. (ed.) GPU Gems 2, pp. 177–187. Addison-Wesley, Reading (2005)

  17. Kautz, J., McCool, M.D.: Approximation of glossy reflection with prefiltered environment maps. In: Graphics Interface, pp. 119–126 (2000)

  18. Keller, A.: Instant radiosity. In: SIGGRAPH ’97: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp 49–56. ACM Press, New York (1997). doi:http://doi.acm.org/10.1145/258734.258769

  19. Klinker, G., Dutoit, A.H., Bauer, M., Bayer, J., Novak, V., Matzke, D.: Fata morgana—a presentation system for product design. In: ISMAR ’02: Proceedings of the 1st International Symposium on Mixed and Augmented Reality, p. 76. IEEE Computer Society, Washington, DC (2002)

  20. Lehtinen, J., Kautz, J.: Matrix radiance transfer. In: I3D ’03: Proceedings of the 2003 Symposium on Interactive 3D Graphics, pp. 59–64. ACM Press, New York (2003). doi:http://doi.acm.org/10.1145/641480.641495

  21. Müller, G., Meseth, J., Klein, R.: Fast environmental lighting for local-pca encoded btfs. In: CGI ’04: Proceedings of the Computer Graphics International, pp. 198–205. IEEE Computer Society, Washington, DC (2004). doi:http://dx.doi.org/10.1109/CGI.2004.49

  22. Nicodemus, F.E., Richmond, J.C., Hsia, J.J., Ginsberg, I.W., Limperis, T.: Geometrical considerations and nomenclature for reflectance. Final Report National Bureau of Standards, Washington, DC (1977)

  23. O’Malley, S.M.: A simple, effective system for automated capture of high dynamic range images. In: ICVS ’06: Proceedings of the Fourth IEEE International Conference on Computer Vision Systems, p. 15. IEEE Computer Society, Washington, DC (2006). doi:http://dx.doi.org/10.1109/ICVS.2006.8

  24. OpenGL Architecture Review Board, Shreiner, D., Woo, M., Neider, J., Davis, T.: OpenGL(R) Programming Guide: The Official Guide to Learning OpenGL(R), Version 2.1. Addison-Wesley, Reading (2007)

  25. OpenSG.: (2009). http://opensg.vrsource.org

  26. Page\Park Architects: (2009). http://www.pagepark.co.uk

  27. pfstools (2009): http://pfstools.sourceforge.net

  28. Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. In: SIGGRAPH ’02: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 267–276. ACM Press, New York (2002). doi:http://doi.acm.org/10.1145/566570.566575

  29. Rost, R.J.: OpenGL(R) Shading Language. Addison-Wesley, Redwood City (2004)

  30. Shirley, P., Morley, R.K.: Realistic Ray Tracing. A. K. Peters Ltd, Natick (2003)

  31. Sillion, F.X., Arvo, J.R., Westin, S.H., Greenberg, D.P.: A global illumination solution for general reflectance distributions. In: SIGGRAPH ’91: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–196. ACM Press, New York (1991). doi:http://doi.acm.org/10.1145/122718.122739

  32. Sloan, P.P., Kautz, J., Snyder, J.: Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In: SIGGRAPH ’02: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 527–536. ACM Press, New York (2002). doi:http://doi.acm.org/10.1145/566570.566612

  33. Spheron: (2009). http://www.spheron.com

  34. Wald, I., Dietrich, A., Benthin, C., Efremov, A., Dahmen, T., Günther, J., Havran, V., Seidel, H.P., Slusallek, P.: Applying ray tracing for virtual reality and industrial design. In: Proceedings of the 2006 IEEE Symposium on Interactive Ray Tracing, pp. 177–185 (2006)

  35. Wallner, G.: GPU radiosity for triangular meshes with support of normal mapping and arbitrary light distributions. J. WSCG 16(1–3), 1–8 (2008)

    Google Scholar 

  36. Whitted, T.: An improved illumination model for shaded display. Commun. ACM 23(6), 343–349 (1980). doi:http://doi.acm.org/10.1145/358876.358882

    Google Scholar 

  37. Williams, L.: Casting curved shadows on curved surfaces. SIGGRAPH Comput. Graph. 12(3), 270–274 (1978). doi:http://doi.acm.org/10.1145/965139.807402

  38. Zhou, K., Hou, Q., Wang, R., Guo, B.: Real-time kd-tree construction on graphics hardware. In: SIGGRAPH Asia’08: ACM SIGGRAPH Asia 2008 papers, pp. 1–11. ACM Press, New York (2008). doi:http://doi.acm.org/10.1145/1457515.1409079

Download references

Acknowledgments

The research leading to these results has received funding from the European Community’s Sixth and Seventh Framework Programme (under grant agreement FP6-IST-2-004785 and FP7-ICT-1-217039).

Author information

Authors and Affiliations

  1. Fraunhofer IGD, Darmstadt, Germany

    Thomas Gierlinger & Daniel Danch

  2. TU Darmstadt, Darmstadt, Germany

    André Stork

Authors
  1. Thomas Gierlinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Danch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. André Stork
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Gierlinger.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gierlinger, T., Danch, D. & Stork, A. Rendering techniques for mixed reality. J Real-Time Image Proc 5, 109–120 (2010). https://doi.org/10.1007/s11554-009-0137-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-009-0137-x

Keywords

Search

Navigation