Skip to main content
SpringerLink
Log in

The Depth Estimation of 2D Content: A New Life for Paintings

  • Conference paper
  • First Online:
Extended Reality (XR Salento 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14219))

Included in the following conference series:

  • 493 Accesses

Abstract

The preservation, accessibility, and dissemination of historical artifacts to a wider audience have become increasingly important, and cultural institutions can achieve these goals through the digitization of cultural heritage. In recent years, artificial intelligence (AI) and machine learning (ML) techniques improve the virtualization of cultural artifacts for interactive experiences. In this work, we present a virtualization pipeline for the cultural heritage domain, focusing specifically on paintings, using AI techniques. We outline the basic workflow, including a thorough description of the comparison of various neural network models and their performance metrics. The proposed method creates an immersive experience for viewers to interact with paintings beyond observation. The approach utilizes 2.5D technology by applying depth maps of paintings using deep learning (DL) algorithms. The proof of concept was demonstrated on two real-life paintings of varying complexities, and this innovative approach holds potential for enhancing the appreciation and understanding of cultural heritage in museums and other cultural institutions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The interactive visualization: https://bit.ly/41oVsWO.

  2. 2.

    Demo: https://ipolcore.ipol.im/demo/clientApp/demo.html?id=459.

  3. 3.

    PTGui official website: https://ptgui.com/.

  4. 4.

    Depth Player official website: https://bit.ly/3nMJsk7),.

References

  1. Depth predictions in art. https://storage.googleapis.com/art/history/depth/data/demo/index.html

  2. Altınbay, R., Gümüş, N.: Social studies teachers’ views on the virtual tour applications. J. Innov. Res. Teacher Educ. 1(1), 60–71 (2020). https://doi.org/10.29329/jirte.2020.321.5

  3. Chen, P.Y., Liu, A.H., Liu, Y.C., Wang, Y.C.F.: Towards scene understanding: unsupervised monocular depth estimation with semantic-aware representation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2619–2627 (2019). https://doi.org/10.1109/CVPR.2019.00273

  4. Chen, T., An, S., Zhang, Y., Ma, C., Wang, H., Guo, X., Zheng, W.: Improving monocular depth estimation by leveraging structural awareness and complementary datasets. CoRR abs/2007.11256 (2020). https://arxiv.org/abs/2007.11256

  5. Ehret, T.: Monocular depth estimation: a review of the 2022 state of the art. Image Processing On Line 13, 38–56 (2023). https://doi.org/10.5201/ipol.2023.459

  6. Elkhuizen, W.S., et al.: Comparison of three 3D scanning techniques for paintings, as applied to Vermeer’s ‘Girl with a Pearl Earring’. Heritage Sci. 7(1), 1–22 (2019). https://doi.org/10.1186/s40494-019-0331-5

    Article  Google Scholar 

  7. Fang, Z., Chen, X., Chen, Y., Van Gool, L.: Towards good practice for CNN-based monocular depth estimation. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1080–1089 (2020). https://doi.org/10.1109/WACV45572.2020.9093334

  8. Furferi, R., Governi, L., Volpe, Y., Puggelli, L., Vanni, N., Carfagni, M.: From 2D to 2.5D i.e. from painting to tactile model. Graph Models 76(6), 706–723 (2014). https://doi.org/10.1016/j.gmod.2014.10.001

  9. Galeazzi, F., Franco, P.D.G.D., Matthews, J.L.: Comparing 2D pictures with 3D replicas for the digital preservation and analysis of tangible heritage. Museum Manage. Curatorship 30(5), 462–483 (2015). https://doi.org/10.1080/09647775.2015.1042515

    Article  Google Scholar 

  10. Giuliani, F., De Paoli, R., Di Miceli, E.: A risk-reduction framework for urban cultural heritage: a comparative study on Italian historic centres. J. Cultural Heritage Manage. Sustain. Dev. 11(4), 499–515 (2021). https://doi.org/10.1108/JCHMSD-07-2020-0099

    Article  Google Scholar 

  11. Guarneri, M., De Collibus, M.F., Francucci, M., Ciaffi, M.: The importance of artworks 3D digitalization at the time of Covid epidemy: case studies by the use of a multi-wavelengths technique. In: 2020 IEEE 5th International Conference on Image, Vision and Computing (ICIVC), pp. 113–117 (2020). https://doi.org/10.1109/ICIVC50857.2020.9177443

  12. Huynh, L., Nguyen-Ha, P., Matas, J., Rahtu, E., Heikkilä, J.: Guiding monocular depth estimation using depth-attention volume. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12371, pp. 581–597. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58574-7_35

    Chapter  Google Scholar 

  13. Jin, S., Fan, M., Wang, Y., Liu, Q.: Reconstructing traditional Chinese paintings with immersive virtual reality. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–8. CHI EA ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3334480.3382934

  14. Johnston, A., Carneiro, G.: Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume, pp. 4755–4764 (2020). https://doi.org/10.1109/CVPR42600.2020.00481

  15. Khan, F., Salahuddin, S., Javidnia, H.: Deep learning-based monocular depth estimation methods-a state-of-the-art review. Sensors 20(8) (2020). https://doi.org/10.3390/s20082272, https://www.mdpi.com/1424-8220/20/8/2272

  16. Kim, J.H., Ko, K.L., Le Ha, T., Jung, S.W.: Monocular depth estimation of old photos via collaboration of monocular and stereo networks. IEEE Access 11, 11675–11684 (2023). https://doi.org/10.1109/ACCESS.2023.3241348

    Article  Google Scholar 

  17. Lee, J.H., Kim, C.S.: Single-image depth estimation using relative depths. J. Vis. Commun. Image Representation 84, 103459 (2022). https://doi.org/10.1016/j.jvcir.2022.103459

    Article  Google Scholar 

  18. Lee, J., Kim, C.S.: Multi-loss rebalancing algorithm for monocular depth estimation, pp. 785–801 (2020). https://doi.org/10.1007/978-3-030-58520-4_46

  19. Liu, Z.S., Wang, L.W., Siu, W.C., Kalogeiton, V.: Name your style: an arbitrary artist-aware image style transfer (2022)

    Google Scholar 

  20. Pan, J., Li, L., Yamaguchi, H., Hasegawa, K., Thufail, F.I., Brahmantara, Tanaka, S.: 3D reconstruction of Borobudur reliefs from 2D monocular photographs based on soft-edge enhanced deep learning. ISPRS J. Photogram. Remote Sens. 183, 439–450 (2022). https://doi.org/10.1016/j.isprsjprs.2021.11.007, https://www.sciencedirect.com/science/article/pii/S0924271621003051

  21. Park, S., Chon, S., Lee, T., Kim, J.: Toward the experiential VR Gallery using 2.5-D. EasyChair Preprint no. 1091 (EasyChair, 2019)

    Google Scholar 

  22. Poornapushpakala, S., Barani, S., Subramoniam, M., Vijayashree, T.: Restoration of Tanjore paintings using segmentation and in-painting techniques. Heritage Sci. 10(1), 1–6 (2022). https://doi.org/10.1186/s40494-022-00661-1

    Article  Google Scholar 

  23. Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer. 44, 1623–1637 (2022). https://doi.org/10.1109/TPAMI.2020.3019967

  24. Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction (2021). https://doi.org/10.48550/ARXIV.2103.13413

  25. Romão, X., Paupério, E.: An indicator for post-disaster economic loss valuation of impacts on cultural heritage. Int. J. Architect. Heritage, 1–20 (2019). https://doi.org/10.1080/15583058.2019.1643948

  26. Skamantzari, M., Georgopoulos, A.: 3D visualization for virtual museum development. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLI-B5, 961–968 (2016). https://doi.org/10.5194/isprs-archives-XLI-B5-961-2016

  27. Tatlı, Z., Çelenk, G., Altınışık, D.: Analysis of virtual museums in terms of design and perception of presence. Educ. Inf. Technol. (2023). https://doi.org/10.1007/s10639-022-11561-z

    Article  Google Scholar 

  28. Torre, S.D.: Italian perspective on the planned preventive conservation of architectural heritage. Front. Architect. Res. 10(1), 108–116 (2021). https://doi.org/10.1016/j.foar.2020.07.008

    Article  Google Scholar 

  29. Wang, C., Lucey, S., Perazzi, F., Wang, O.: Web stereo video supervision for depth prediction from dynamic scenes (2019). https://doi.org/10.48550/arxiv.1904.11112

  30. Wang, L., Zhang, J., Wang, O., Lin, Z., Lu, H.: SDC-Depth: semantic divide-and-conquer network for monocular depth estimation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 538–547 (2020). https://doi.org/10.1109/CVPR42600.2020.00062

  31. Wang, L., Zhang, J., Wang, Y., Lu, H., Ruan, X.: CLIFFNet for monocular depth estimation with hierarchical embedding loss, pp. 316–331 (10 2020). https://doi.org/10.1007/978-3-030-58558-7_19

  32. Xian, K., Zhang, J., Wang, O., Mai, L., Lin, Z., Cao, Z.: Structure-guided ranking loss for single image depth prediction. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  33. Xue, A.: End-to-end Chinese landscape painting creation using generative adversarial networks (2020)

    Google Scholar 

  34. Yin, W., Liu, Y., Shen, C., Yan, Y.: Enforcing geometric constraints of virtual normal for depth prediction, pp. 5683–5692 (10 2019). https://doi.org/10.1109/ICCV.2019.00578

  35. Yin, W., Liu, Y., Shen, C., Yan, Y.: Enforcing geometric constraints of virtual normal for depth prediction (2019). https://doi.org/10.48550/ARXIV.1907.12209

  36. Yin, W., et al.: Learning to recover 3D scene shape from a single image (2020). https://doi.org/10.48550/arxiv.2012.09365

  37. Zamir, A., Sax, A., Shen, W., Guibas, L., Malik, J., Savarese, S.: Taskonomy: disentangling task transfer learning (2018). https://doi.org/10.48550/arxiv.1804.08328

Download references

Author information

Authors and Affiliations

  1. Vision Robotics and Artificial Intelligence (VRAI) lab, Department of Information Engineering, Marche Polytechnic University, Ancona, Italy

    Aleksandra Pauls, Roberto Pierdicca, Adriano Mancini & Primo Zingaretti

Authors
  1. Aleksandra Pauls
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roberto Pierdicca
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adriano Mancini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Primo Zingaretti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aleksandra Pauls .

Editor information

Editors and Affiliations

  1. University of Salento, Lecce, Italy

    Lucio Tommaso De Paolis

  2. University of Naples Federico II, Naples, Italy

    Pasquale Arpaia

  3. CNR-STIIMA, Lecco, Italy

    Marco Sacco

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pauls, A., Pierdicca, R., Mancini, A., Zingaretti, P. (2023). The Depth Estimation of 2D Content: A New Life for Paintings. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2023. Lecture Notes in Computer Science, vol 14219. Springer, Cham. https://doi.org/10.1007/978-3-031-43404-4_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43404-4_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43403-7

  • Online ISBN: 978-3-031-43404-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation