Skip to main content
SpringerLink
Log in

Machines Learning for Mixed Reality

The Milan Cathedral from Survey to Holograms

  • Conference paper
  • First Online:
Pattern Recognition. ICPR International Workshops and Challenges (ICPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12663))

Included in the following conference series:

  • 2670 Accesses

Abstract

In recent years, a complete 3D mapping of the Cultural Heritage (CH) has become fundamental before every other action could follow. Different survey techniques outputs could be combined in a 3D point cloud, completely describing the geometry of even the most complex object. These data very rich in metric quality can be used to extract 2D technical elaborations and advanced 3D representations to support conservation interventions and maintenance planning.

The case of Milan Cathedral is outstanding. In the last 12 years, a multi-technique omni-comprehensive survey has been carried out to extract the technical representations that are used by the Veneranda Fabbrica (VF) del Duomo di Milano to plan its maintenance and conservation activities.

Nevertheless, point cloud data lack structured information such as semantics and hierarchy among parts, fundamentals for 3D model interaction and database (DB) retrieval. In this context, the introduction of point cloud classification methods could improve data usage, model definition and analysis.

In this paper, a Multi-level Multi-resolution (MLMR) classification approach is presented and tested on the large dataset of Milan Cathedral. The 3D point model, so structured, for the first time, is used directly in a Mixed Reality (MR) environment to develop an application that could benefit professional works, allowing to use 3D survey data on-site, supporting VF activities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Conducted by Politecnico di Milano, ABC department, Group 3Dsurvey.

  2. 2.

    “The “Veneranda Fabbrica del Duomo di Milano” is an ecclesiastical body endowed with legal personality by ancient statutory determination with the purpose of worship and religion, which excludes all profit-making activities. Its earliest regulations were issued on 16 October 1387 at the behest of Gian Galeazzo Visconti.” [1]

  3. 3.

    The HoloLens 2 device works on the possibility to track hands movements with its cameras, when it recognizes that an hand is present in the rendering area it will project a dashed line in the direction to which the hand is pointing.

  4. 4.

    Air tap interaction consists in touching the index finger with the thumb of the same hand and immediately releasing them while the HoloLens 2 is tracking their position.

References

  1. Veneranda Fabbrica Homepage. https://www.duomomilano.it/en/infopage/veneranda-fabbrica-del-duomo-di-milano/68/. Accessed 21 Oct 2020

  2. Fassi, F., Achille, C., Fregonese, L.: Surveying and modelling the main spire of milan cathedral using multiple data sources. Photogram. Rec. 26, 462–487 (2011)

    Article  Google Scholar 

  3. Fassi, F., Achille, C., Mandelli, A., Rechichi, F., Parri, S.: A new idea of BIM system for visualization, WEB sharing and using huge complex 3D models for facility management. In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XL-5/W4, pp. 359–366 (2015)

    Google Scholar 

  4. Rechichi, F., Mandelli, A., Achille, C., Fassi, F.: Sharing high-resolution modules and information on the WEB: the WEB module of BIM3DSG system. In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLI-B5, pp. 703–710 (2016)

    Google Scholar 

  5. Fassi, F., Parri, S.: Complex architecture in 3D: from survey to web. Int. J. Heritage Digit. Era 1(3), 379–398 (2012)

    Article  Google Scholar 

  6. Achille, C., Fassi, F., Mandelli, A., Perfetti, L., Rechichi, F., Teruggi, S.: From a traditional to a digital site: 2008–2019. The history of Milan Cathedral surveys. In: Daniotti, B., Gianinetto, M., Della Torre, S. (eds.) Digital Transformation of the Design, Construction and Management Processes of the Built Environment. RD, pp. 331–341. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-33570-0_30

    Chapter  Google Scholar 

  7. Perfetti, L., Fassi, F., Gulsan, H.: Generation of gigapixel orthophoto for the maintenance of complex buildings. Challenges and lesson learnt. In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLII-2/W9, pp. 605–614 (2019)

    Google Scholar 

  8. Mandelli, A., Fassi, F., Perfetti, L., Polari, C.: Testing different survey techniques to model architectonic narrow spaces. In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLII-2/W5, pp. 505–511 (2017)

    Google Scholar 

  9. Teruggi, S., Grilli, E., Russo, M., Fassi, F., Remondino, F.: A hierarchical machine learning approach for multi-level and multi-resolution 3D point cloud classification. Remote Sens. 12(16), 2598 (2020)

    Article  Google Scholar 

  10. Reboli, D., Pučko, Z., Babič, N.Č, Bizjak, M., Mongus, D.: Point cloud quality requirements for scan-vs-BIM based automated construction progress monitoring. Autom. Constr. 84, 323–334 (2017)

    Article  Google Scholar 

  11. Son, H., Kim, C.: Semantic as-built 3D modelling of structural elements of buildings based in local concavity and convexity. Adv. Eng. Inform. 34, 114–124 (2017)

    Article  Google Scholar 

  12. Bassier, M., Yousefzadeh, M., Vergauwen, M.: Comparison of 2D and 3D wall reconstruction algorithms from point cloud data for as-built BIM. J. Inf. Technol. Constr. (ITcon) 25(11), 173–192 (2020)

    Google Scholar 

  13. Apollonio, F.I., et al.: A 3D-centered information system for the documentation of a complex restoration intervention. J. Cult. Heritage 29, 89–99 (2018)

    Article  Google Scholar 

  14. Croce, V., Caroti, G., De Luca, L., Piemonte, A., Véron, P.: Semantic annotation of heritage models: 2D/3D approaches and future research challenges. In: The International Archives of Photogrammetry, Remote Sensing and Spatial Information Science, vol. 43, pp. 829–836 (2020)

    Google Scholar 

  15. Sánchez-Aparicio, L.J., Del Pozo, S., Ramos, L.F., Arce, A., Fernandes, F.M.: Heritage site preservation with combined radiometric and geometric analysis of TLS data. Automation in Construction 85, 24–39 (2018)

    Article  Google Scholar 

  16. Valero, E., Bosché, F., Forster, A.: Automatic segmentation of 3D point cloud of rubble masonry walls and its application to building surveying Repair and Maintenance. Autom.Constr. 96, 29–39 (2018)

    Article  Google Scholar 

  17. Roussel, R., Bagnéris, M., De Luca, L., Bomblet, P.: A digital diagnosis for the <<Autumn>> statue (Marseille, France): photogrammetry, digital cartography and construction of a thesaurus. In: International Archives of Photogrammetry, Remote Sensing and Spatial Information Science, vol. XLII-2/W15, pp. 1039–1046 (2019)

    Google Scholar 

  18. Mizoguchi, T., et al.: Quantitative scaling evaluation of concrete structures based on terrestrial laser scanning. Autom. Constr. 35, 263–274 (2013)

    Article  Google Scholar 

  19. Kashani, A.G., Graettinger, A.J.: Cluster-based roof covering damage detection in ground-based lidar data. Autom. Constr. 58, 19–27 (2015)

    Article  Google Scholar 

  20. Matrone, F., et al.: A benchmark for large-scale heritage point cloud semantic segmentation. In: International Archives of Photogrammetry, Remote Sensing and Spatial Information Science, vol. XLIII-B2, pp. 1419–1426 (2020)

    Google Scholar 

  21. Murtiyoso, A., Grussenmeyer, P.: Virtual disassembling of historical edifices: experiments and assessments of an automatic approach for classifying multi-scalar point clouds into architectural elements. Sensors 20(8), 2161 (2020)

    Article  Google Scholar 

  22. Pierdicca, R., et al.: Point cloud semantic segmentation using deep learning framework for cultural heritage. Remote Sens. 12(6), 1005 (2020)

    Article  Google Scholar 

  23. Grilli, E., Dininno, D., Marsicano, L., Petrucci, G., Remondino, F.: Supervised segmentation of 3D cultural heritage. In 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) Held Jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), San Francisco, CA, USA, pp. 1–8. IEEE (2018)

    Google Scholar 

  24. Grilli, E., M. Farella, E., Torresani, A., Remondino, F.: geometric features analysis for the classification of cultural heritage point clouds. In: International Archives of Photogrammetry, Remote Sensing and Spatial Information Science, vol. XLII-2/W15, pp. 541–548 (2019)

    Google Scholar 

  25. Grilli, E., Remondino, F.: Machine learning generalization across different 3D architectural heritage. ISPRS Int. J. Geo-Inf. 9(6), 379 (2020)

    Article  Google Scholar 

  26. Matrone, F., Grilli, E., Martini, M., Paolanti, M., Pierdicca, R., Remondino, F.: Comparing machine and deep learning methods for large 3D heritage semantic segmentation. ISPRS Int. J. Geo-Inf. 9(9), 535 (2020)

    Article  Google Scholar 

  27. Blomley, R., Weinmann, M., Leitloff, J., Jutzi, B.: Shape distribution features for point cloud analysis – a geometric histogram approach on multiple scales. In: ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 2, no. 3, p. 9 (2014)

    Google Scholar 

  28. Breimann, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  29. Goutte, C., Gaussier, E.: A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 345–359. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31865-1_25

    Chapter  Google Scholar 

  30. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipulator Telepresence Technol. 2351, 282–292 (1995)

    Article  Google Scholar 

  31. De Pace, F., Manuri, F., Sanna, A.: Augmented reality in Industry 4.0. Am. J. Comput. Sci. Inf. Technol. 06(01), 1–7 (2018)

    Google Scholar 

  32. Fraga-Lamas, P., Fernandez-Carames, T. M., Blanco-Nova, O., Vilar-Montesinos, M. A.: A review of industrial augmented reality systems for the Industry 4.0 Shipyard. IEEE Access 6, 13358–13375 (2018)

    Google Scholar 

  33. Desselle, M.R., Brown, R.A., James, A.R., Midwinter, M.J., Powel, S.K., Woodruff, M.A.: Augmented and virtual reality in surgery. Comput. Sci. Eng. 22(3), 18–26 (2020)

    Article  Google Scholar 

  34. Blanco-Pons, S., Carrion-Ruiz, B., Duong, M., Chartrand, J., Fai, S., Luis Lerma, J.: Augmented reality markerless multi-image outdoor tracking system for the historical buildings on parliament hill. Sustainability 11(16), 4268 (2019)

    Article  Google Scholar 

  35. Unity 3D Homepage. https://unity.com/. Accessed 22 Oct 2020

  36. MRTKv2 GitHub page. https://github.com/microsoft/MixedRealityToolkit-Unity. Accessed 22 Oct 2020

  37. HoloLens 2 Homepage. https://www.microsoft.com/it-it/hololens/hardware. Accessed 22 Oct 2020

Download references

Acknowledgement

The authors would like to thank all the colleagues who have participated in the past and are now collaborating in the Milan Cathedral project. A special thanks to Ing. Francesco Canali, yard director of “Veneranda Fabbrica del Duomo di Milano”. Thanks to the colleagues of FBK of Trento 3DOM, Fabio Remondino and Eleonora Grilli with whom the research on the ML classification of the Cathedral of Milan has been carried out and more in-depth presented in [9].

Author information

Authors and Affiliations

  1. 3DSurveyGroup-ABCLab-Politecnico di Milano, Via Ponzio 31, 20133, Milan, Italy

    Simone Teruggi & Francesco Fassi

Authors
  1. Simone Teruggi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francesco Fassi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simone Teruggi .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Alberto Del Bimbo

  2. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Rita Cucchiara

  3. Department of Computer Science, Boston University, Boston, MA, USA

    Stan Sclaroff

  4. Dipartimento di Matematica e Informatica, University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Cloud & AI, JD.COM, Beijing, China

    Tao Mei

  6. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Marco Bertini

  7. Computational Sciences Department, National Institute of Astrophysics, Optics and Electronics (INAOE), Tonantzintla, Puebla, Mexico

    Hugo Jair Escalante

  8. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Roberto Vezzani

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Teruggi, S., Fassi, F. (2021). Machines Learning for Mixed Reality. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12663. Springer, Cham. https://doi.org/10.1007/978-3-030-68796-0_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68796-0_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68795-3

  • Online ISBN: 978-3-030-68796-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation