Abstract
The joint usage of Extended Reality (XR) and Artificial Intelligence (AI) has enabled different Metaverse-related use cases. Such paradigms were recently adopted for immersive content creation, particularly considering Neural Rendering (NR) techniques to project scenes from the real world in the 3D realm. These methods are particularly beneficial in the field of Cultural Heritage (CH), where digitizing and visualizing cultural assets in 3D is crucial. However, current evaluation protocols lack a robust integration of human judgments through a Human-In-The-Loop (HITL) approach to humanly evaluate the quality of the generated 3D models, which could also support model optimization. To bridge this gap, we here introduce X-NR, a novel XR framework designed to evaluate and compare 3D reconstruction methodologies, including NR in the context of CH. We contextualize and validate such a framework through case studies on cultural heritage sites in the Marche region (Italy), employing various data-capturing and 3D reconstruction methodologies. The study concludes with a validation of the framework by CH domain experts, underscoring its potential advantages over traditional 3D editing software.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Achiam, J., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
Balloni, E., Gorgoglione, L., Paolanti, M., Mancini, A., Pierdicca, R.: Few shot photogrametry: a comparison between nerf and mvs-sfm for the documentation of cultural heritage. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 48, 155–162 (2023)
Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability Evaluation Industry 189(194), 4–7 (1996)
Chen, M., Jin, Y., Goodall, T., Yu, X., Bovik, A.C.: Study of 3d virtual reality picture quality. IEEE J. Selected Topics Signal Process. 14(1), 89–102 (2019)
Croce, V., Caroti, G., De Luca, L., Piemonte, A., Véron, P.: Neural radiance fields (nerf): review and potential applications to digital cultural heritage. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 48, 453–460 (2023)
Deng, C., et al.: Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20637–20647 (2023)
European Commission: Cultural heritage and cultural and creative industries (ccis). https://research-and-innovation.ec.europa.eu/research-area/social-sciences-and-humanities/cultural-heritage-and-cultural-and-creative-industries-ccis_en (Year of access), Accessed 17 March 2024
Fangi, G., Wahbeh, W., Malinverni, E.S., Di Stefano, F., Pierdicca, R.: Documentation of syrian lost heritage: From 3d reconstruction to open information system (2022)
Gao, K., Gao, Y., He, H., Lu, D., Xu, L., Li, J.: Nerf: Neural radiance field in 3d vision, a comprehensive review. arXiv preprint arXiv:2210.00379 (2022)
Guédon, A., Lepetit, V.: Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 (2023)
Hart, S.G., Staveland, L.E.: Development of nasa-tlx (task load index): results of empirical and theoretical research. In: Advances in psychology, vol. 52, pp. 139–183. Elsevier (1988)
Hily, A., Dupont, L., Arbelaez-Garces, G., Camargo, M., Dinet, J.: Evaluation and validation process of extended reality applications developed in an industrial context: a systematic review. SN Comput. Sci. 4(5), 637 (2023)
Hirzle, T., Müller, F., Draxler, F., Schmitz, M., Knierim, P., Hornbæk, K.: When xr and ai meet-a scoping review on extended reality and artificial intelligence. In: Proceedings of the 2023 CHI Conference on Human Factors in ComputingmSystems, pp. 1–45 (2023)
Holzwarth, V., Gisler, J., Hirt, C., Kunz, A.: Comparing the accuracy and precision of steamvr tracking 2.0 and oculus quest 2 in a room scale setup. In: Proceedings of the 2021 5th International Conference on Virtual and Augmented Reality Simulations, pp. 42–46 (2021)
Huynh-The, T., Pham, Q.V., Pham, X.Q., Nguyen, T.T., Han, Z., Kim, D.S.: Artificial intelligence for the metaverse: a survey. Eng. Appl. Artif. Intell. 117, 105581 (2023)
Jin, Y., Chen, M., Goodall, T., Patney, A., Bovik, A.C.: Subjective and objective quality assessment of 2d and 3d foveated video compression in virtual reality. IEEE Trans. Image Process. 30, 5905–5919 (2021)
Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4) (2023)
Li, B., Xu, X., Tang, S., Yu, L., Wang, Z.: Human perception-guided meta-training for few-shot nerf. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13206–13210. IEEE (2024)
Li, K., Schmidt, S., Rolff, T., Bacher, R., Leemans, W., Steinicke, F.: Magic nerf lens: interactive fusion of neural radiance fields for virtual facility inspection. Front. Virtual Reality 5, 1377245 (2024)
Li, Z., et al.: Neuralangelo: high-fidelity neural surface reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8456–8465 (2023)
Li, Z., Zhang, Y., Wu, C., Zhu, J., Zhang, L.: Ho-gaussian: Hybrid optimization of 3d gaussian splatting for urban scenes. arXiv preprint arXiv:2403.20032 (2024)
Liang, H., Wu, T., Hanji, P., Banterle, F., Gao, H., Mantiuk, R., Oztireli, C.: Perceptual quality assessment of nerf and neural view synthesis methods for front-facing views. arXiv preprint arXiv:2303.15206 (2023)
Long, X., et al.: Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 (2023)
Martin, P., Rodrigues, A., Ascenso, J., Queluz, M.P.: Nerf-qa: neural radiance fields quality assessment database. In: 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), pp. 107–110. IEEE (2023)
Mazzacca, G., et al.: Nerf for heritage 3d reconstruction. Inter. Archiv. Photogrammetry, Remote Sensing Spatial Inform. Sci. 48(M-2-2023), 1051–1058 (2023)
Meng, X., Chen, W., Yang, B.: Neat: learning neural implicit surfaces with arbitrary topologies from multi-view images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248–258 (2023)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24
Nagta, A., Sharma, B., Sharma, A., et al.: Oculus: a new dimension to virtual reality. In: 2022 International Conference on Automation, Computing and Renewable Systems (ICACRS), pp. 1169–1172. IEEE (2022)
Nehmé, Y., Dupont, F., Farrugia, J.P., Le Callet, P., Lavoué, G.: Visual quality of 3d meshes with diffuse colors in virtual reality: subjective and objective evaluation. IEEE Trans. Visual Comput. Graph. 27(3), 2202–2219 (2020)
Onuoha, C., Flaherty, J.A., Luo, S., Huong, T.T., Thang, T.C.: An evaluation of quality metrics for neural radiance field. In: 2023 IEEE 15th International Conference on Computational Intelligence and Communication Networks (CICN), pp. 619–623. IEEE (2023)
Pepe, M., Alfio, V.S., Costantino, D.: Assessment of 3d model for photogrammetric purposes using ai tools based on nerf algorithm. Heritage 6(8), 5719–5731 (2023)
Perez-Ortiz, M., Mikhailiuk, A., Zerman, E., Hulusic, V., Valenzise, G., Mantiuk, R.K.: From pairwise comparisons and rating to a unified quality scale. IEEE Trans. Image Process. 29, 1139–1151 (2019)
Picardi, A., Caruso, G.: User-centered evaluation framework to support the interaction design for augmented reality applications. Multimodal Technol. Interact. 8(5), 41 (2024)
Qu, Q., Liang, H., Chen, X., Chung, Y.Y., Shen, Y.: Nerf-nqa: no-reference quality assessment for scenes generated by nerf and neural view synthesis methods. IEEE Trans. Visualizat. Comput. Graph. (2024)
Qureshi, A.H., Alaloul, W.S., Murtiyoso, A., Saad, S., Manzoor, B.: Comparison of photogrammetry tools considering rebar progress recognition. Intern. Archiv. Photogrammetry, Remote Sensing Spatial Inform. Sci. 43, 141–146 (2022)
Scorolli, C., Grasso, E.N., Stacchio, L., Armandi, V., Matteucci, G., Marfia, G.: Would you rather come to a tango concert in theater or in vr? aesthetic emotions & social presence in musical experiences, either live, 2d or 3d. Computers in Human Behavior, p. 107910 (2023)
Series, B.: Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT 500(13) (2012)
Speicher, M., Cucerca, S., Krüger, A.: Vrshop: a mobile interactive virtual reality shopping environment combining the benefits of on-and offline shopping. Proc. ACM Interact. Mobile, Wearable Ubiquitous Technol. 1(3), 1–31 (2017)
Stacchio, L., Scorolli, C., Marfia, G.: Evaluating human aesthetic and emotional aspects of 3d generated content through extended reality (2022)
Tancik, M., et al.: Nerfstudio: a modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–12 (2023)
Teruggi, S., Grilli, E., Fassi, F., Remondino, F.: 3d surveying, semantic enrichment and virtual access of large cultural heritage. ISPRS Annals Photogrammetry, Remote Sensing Spatial Inform. Sci. 8, 155–162 (2021)
Tewari, A., et al.: State of the art on neural rendering. In: Computer Graphics Forum, vol. 39, pp. 701–727. Wiley Online Library (2020)
Tosi, F., et al.: How nerfs and 3d gaussian splatting are reshaping slam: a survey. arXiv preprint arXiv:2402.13255 (2024)
Venkatesh, V., Bala, H.: Technology acceptance model 3 and a research agenda on interventions. Decis. Sci. 39(2), 273–315 (2008)
Wang, Y., et al.: A survey on metaverse: Fundamentals, security, and privacy. IEEE Commun. Surv. Tutorials (2022)
Yu, Z., et al.: Sdfstudio: A unified framework for surface reconstruction (2022)
Zerman, E., Hulusic, V., Valenzise, G., Mantiuk, R.K., Dufaux, F.: The relation between mos and pairwise comparisons and the importance of cross-content comparisons. Electronic Imaging 30, 1–6 (2018)
Acknowledgements
This work has been funded by the European Union - NextGenerationEU under the Italian Ministry of University and Research (MUR) National Innovation Ecosystem grant ECS00000041 - VITALITY - CUP D83C22000710005.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Stacchio, L., Balloni, E., Gorgoglione, L., Paolanti, M., Frontoni, E., Pierdicca, R. (2024). X-NR: Towards An Extended Reality-Driven Human Evaluation Framework for Neural-Rendering. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2024. Lecture Notes in Computer Science, vol 15027. Springer, Cham. https://doi.org/10.1007/978-3-031-71707-9_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-71707-9_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71706-2
Online ISBN: 978-3-031-71707-9
eBook Packages: Computer ScienceComputer Science (R0)