Skip to main content
SpringerLink
Log in

Design and calibration of curved and see-through integral imaging 3D display

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Heads-up displays that are ‘see-through’ and ‘curved’ and capable of displaying 3D contents are considered crucial for augmented reality-based navigation in automobiles. Here we report the development, calibration and experimental evaluation of a 3D display system that satisfies the above requirements. Integral imaging is used as the 3D display technique, which is realized using a flexible ‘concave-micro-mirror array’ screen (equivalent of a ‘micro-lens array’, but working in reflection mode). The screen itself is fabricated as a holographic optical element. The holographic nature of the screen enables a ‘see-through’ effect. The 3D content to be displayed is served by a 2D projector as integral images. A novel calibration method is developed which employs diffusive markers, that are invisible to the naked eye, being placed at one corner of each elemental micro-mirror. The calibration enables proper treatment of the effects and artifacts caused by screen ‘curvature’, but the presence of markers itself does not degrade the display characteristics. A curved micro-mirror array screen of size 10 cm \(\times \) 10 cm consisting of 100 \(\times \) 200 elemental concave mirrors is fabricated as a flexible holographic optical element with diffusive markers of size 300 \(\upmu \)m \(\times \) 300 \(\upmu \)m. The screen, when illuminated with a projector (that serves integral images), was able to reconstruct a 3D scene of size 10 cm \(\times \) 10 cm with a depth of 5 cm. The novel calibration method employing diffusive markers demonstrates significant improvement in calibration accuracy. The curved and see-through nature of the display screen makes it a good choice for windshield displays. The reported system requires further improvements in enlarging the screen size and increasing depth of the 3D scene in order to meet real-world requirements, which can be achieved by scaling-up the system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  • Alizadehsalehi S, Yitmen I (2021) Digital twin-based progress monitoring management model through reality capture to extended reality technologies (DRX). Smart Sustain Built Environ. https://doi.org/10.1108/SASBE-01-2021-0016

  • Alizadehsalehi S, Hadavi A, Huang JC (2020) From bim to extended reality in aec industry. Autom Constr 116:103254

    Article  Google Scholar 

  • Bach D With their HoloLens 2 project, Microsoft and Volkswagen collaborate to put augmented reality glasses in motion. https://news.microsoft.com/transform/with-their-hololens-2-project-microsoft-and-volkswagen-collaborate-to-put-augmented-reality-glasses-in-motion/. Accessed: 2022 July, 25

  • Bang K, Jang C, Lee B (2019) Curved holographic optical elements and applications for curved see-through displays. J Inf Display 20(1):9–23. https://doi.org/10.1080/15980316.2019.1570978

    Article  Google Scholar 

  • Blender Online Community: Blender (2021) a 3D Modelling and Rendering Package. Blender Foundation, Blender Institute, Amsterdam. Blender Foundation. http://www.blender.org

  • Çöltekin A, Lochhead I, Madden M, Christophe S, Devaux A, Pettit C, Lock O, Shukla S, Herman L, Stachoň Z et al (2020) Extended reality in spatial sciences: a review of research challenges and future directions. ISPRS Int J Geo Inf 9(7):439

    Article  Google Scholar 

  • Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  • Hartley R, Zisserman A (2003) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, New York

    MATH  Google Scholar 

  • Hong K, Yeom J, Jang C, Hong J, Lee B (2013) Full-color lens-array holographic optical element for three-dimensional optical see-through augmented reality. Opt Lett 39(1):127. https://doi.org/10.1364/ol.39.000127

    Article  Google Scholar 

  • Huang FC, Luebke DP, Wetzstein G (2015) The light field stereoscope. In: SIGGRAPH emerging technologies, pp 24–1

  • Hyun J-B, Hwang D-C, Shin D-H, Kim E-S (2007) Curved computational integral imaging reconstruction technique for resolution-enhanced display of three-dimensional object images. Appl Opt 46(31):7697–7708. https://doi.org/10.1364/AO.46.007697

    Article  Google Scholar 

  • Jackin BJ, Jorissen L, Oi R, Wu JY, Wakunami K, Okui M, Ichihashi Y, Bekaert P, Huang YP, Yamamoto K (2018) Digitally designed holographic optical element for light field displays. Opt Lett 43(15):3738–3741. https://doi.org/10.1364/OL.43.003738

    Article  Google Scholar 

  • Jang C, Lee C-K, Jeong J, Li G, Lee S, Yeom J, Hong K, Lee B (2015) Recent progress in see-through three-dimensional displays using holographic optical elements [invited]. Appl Opt 55(3):71. https://doi.org/10.1364/ao.55.000a71

    Article  Google Scholar 

  • Javidi B, Carnicer A, Arai J, Fujii T, Hua H, Liao H, Martínez-Corral M, Pla F, Stern A, Waller L, Wang Q-H, Wetzstein G, Yamaguchi M, Yamamoto H (2020) Roadmap on 3d integral imaging: sensing, processing, and display. Opt Express 28(22):32266–32293. https://doi.org/10.1364/OE.402193

    Article  Google Scholar 

  • Jorissen L, Jackin BJ, Oi R, Wakunami K, Okui M, Ichihashi Y, Lafruit G, Yamamoto K, Bekaert P (2019) Homography based identification for automatic and robust calibration of projection integral imaging displays. Appl Opt 58(4):1200–1209

    Article  Google Scholar 

  • Jorissen L, Oi R, Wakunami K, Ichihashi Y, Lafruit G, Yamamoto K, Bekaert P, Jackin BJ (2020) Holographic micromirror array with diffuse areas for accurate calibration of 3d light-field display. Appl Sci 10(20):7188

    Article  Google Scholar 

  • Kawakita M, Sasaki H, Arai J, Okui M, Okano F, Haino Y, Yoshimura M, Sato M (2010) Projection-type integral 3-d display with distortion compensation. J Soc Inform Display 18(9):668–677

    Article  Google Scholar 

  • Kim Y, Park J-H, Choi H, Jung S, Min S-W, Lee B (2004) Viewing-angle-enhanced integral imaging system using a curved lens array. Opt Express 12(3):421–429. https://doi.org/10.1364/OPEX.12.000421

    Article  Google Scholar 

  • Kim Y, Park J-H, Min S-W, Jung S, Choi H, Lee B (2005) Wide-viewing-angle integral three-dimensional imaging system by curving a screen and a lens array. Appl Opt 44(4):546–552. https://doi.org/10.1364/AO.44.000546

    Article  Google Scholar 

  • Li W, Wang H, Zhou M, Wang S, Jiao S, Mei X, Hong T, Lee H, Kim J (2013) Principal observation ray calibration for tiled-lens-array integral imaging display. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1019–1026

  • Lourakis MI, Argyros AA (2009) Sba: a software package for generic sparse bundle adjustment. ACM Trans Math Softw (TOMS) 36(1):1–30

    Article  MathSciNet  MATH  Google Scholar 

  • Martínez-Corral M, Javidi B (2018) Fundamentals of 3d imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems. Adv. Opt. Photon. 10(3):512–566. https://doi.org/10.1364/AOP.10.000512

    Article  Google Scholar 

  • Matusik W, Pfister H (2004) 3d tv: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans Graph (TOG) 23(3):814–824

    Article  Google Scholar 

  • Moreno D, Taubin G (2012) Simple, accurate, and robust projector-camera calibration. In: 2012 Second international conference on 3D imaging, modeling, processing, visualization and transmission. IEEE, pp 464–471

  • Nakamura T, Yamaguchi M (2017) Rapid calibration of a projection-type holographic light-field display using hierarchically upconverted binary sinusoidal patterns. Appl Opt 56(34):9520–9525

    Article  Google Scholar 

  • Newsroom V From the luxury class to the compact segment: the augmented reality head-up display. https://www.volkswagen-newsroom.com/en/press-releases/from-the-luxury-class-to-the-compact-segment-the-augmented-reality-head-up-display-6730/. Accessed 2022 July, 25

  • Sabel T, Lensen MC (2017) Volume holography: novel materials, methods and applications. In: Naydenova I, Nazarova D, Babeva T (eds) Holographic materials and optical systems. IntechOpen, Rijeka . Chaphter 1. https://doi.org/10.5772/67001

  • Shin D-H, Lee B, Kim E-S (2006) Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens. Appl Opt 45(28):7375–7381. https://doi.org/10.1364/AO.45.007375

    Article  Google Scholar 

  • Triviño-Tarradas P, Mohedo Gatón A, Carranza Cañadas P, Burgos-Ladrón de Guevara E, Mesas-Carrascosa FJ, Hidalgo Fernandez RE (2021) Methodology for the virtualisation of engineering drawing exercises for use through extended reality. In: International conference on the digital transformation in the graphic engineering. Springer, pp 423–432

  • Wakunami K, Hsieh P-Y, Oi R, Senoh T, Sasaki H, Ichihashi Y, Okui M, Huang Y-P, Yamamoto K (2016) Projection-type see-through holographic three-dimensional display. Nat Commun 7:12954. https://doi.org/10.1038/ncomms12954

    Article  Google Scholar 

  • Wang W, Chen G, Weng Y, Weng X, Zhou X, Wu C, Guo T, Yan Q, Lin Z, Zhang Y (2020) Large-scale microlens arrays on flexible substrate with improved numerical aperture for curved integral imaging 3D display. Sci Rep 10(1):11741. https://doi.org/10.1038/s41598-020-68620-z

    Article  Google Scholar 

  • Xiao X, Javidi B, Martinez-Corral M, Stern A (2013) Advances in three-dimensional integral imaging: sensing, display, and applications (invited). Appl Opt 52(4):546–560. https://doi.org/10.1364/AO.52.000546

    Article  Google Scholar 

  • Yamaguchi M, Higashida R (2016) 3d touchable holographic light-field display. Appl Opt 55(3):178. https://doi.org/10.1364/ao.55.00a178

    Article  Google Scholar 

  • Yang R, Huang X, Li S, Jaynes C (2007) Toward the light field display: autostereoscopic rendering via a cluster of projectors. IEEE Trans Visual Comput Graphics 14(1):84–96

    Article  Google Scholar 

Download references

Funding

This research was funded by Japan Society For Promotion Of Science (JSPS) Grant No. 18H03281.

Author information

Author notes

  1. Boaz Jessie Jackin and Lode Jorissen have contributed equally to this work.

Authors and Affiliations

  1. Center for Design Centric Engineering, Kyoto Institute of Technology, Matsugasaki, Sakyo, Kyoto, 606-8585, Japan

    Boaz Jessie Jackin

  2. Expertise Centre for Digital Media, Hasselt University-tUL-Flanders Make, Wetenschapspark 2, 3590, Diepenbeek, Belgium

    Lode Jorissen & Philippe Bekaert

  3. Applied Electromagnetic Research Center, National Institute of Information and Communications Technology, 4-2-1 Nukui-Kitamachi, Koganei, Tokyo, 184-8795, Japan

    Ryutaro Oi, Koki Wakunami & Yasuyuki Ichihashi

  4. Faculty of Engineering, Tokushima University, 2-1 Minamijousanjima, Tokushima, Tokushima, 770-8506, Japan

    Kenji Yamamoto

  5. Laboratory of Image Synthesis and Analysis (LISA), Université Libre de Bruxelles/Brussels University, Av. F.D. Roosevelt 50 CP165/57, 1050, Brussels, Belgium

    Gauthier Lafruit

Authors
  1. Boaz Jessie Jackin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lode Jorissen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ryutaro Oi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Koki Wakunami
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kenji Yamamoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yasuyuki Ichihashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Philippe Bekaert
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Gauthier Lafruit
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Boaz Jessie Jackin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This paper is dedicated to the memory of Prof. Dr. Philippe Bekaert.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 0 KB)

Supplementary file 3 (mp4 0 KB)

Supplementary file 2 (mp4 0 KB)

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jackin, B.J., Jorissen, L., Oi, R. et al. Design and calibration of curved and see-through integral imaging 3D display. Virtual Reality 27, 761–775 (2023). https://doi.org/10.1007/s10055-022-00686-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-022-00686-8

Keywords

Search

Navigation