Walkable Mixed Reality Map as interaction interface for Virtual Heritage

https://doi.org/10.1016/j.daach.2019.e00127Get rights and content

Abstract

Studies in the Virtual Heritage (VH) domain have led to underlining the significance of a contextual relationship between users, immersive reality technologies and interactive and engaging cultural context as essential aspects towards enabling cultural learning in VH applications. Interaction methods, therefore, play a great role in terms of enabling interactive and engaging experience under various VH settings. This paper proposes a novel approach to use maps as interaction interfaces in a Mixed Reality (MxR) environment that could be applied to specific VH settings with a predefined cultural and historical context. The primary focus of the proposed interaction method named ‘Walkable MxR Map’ is to use interactive, immersive and walkable maps to allow users interact with cultural content, 3D models, and different multimedia content at museums and heritage sites. Hence, the applicability of the proposed systems will not be limited to museums' indoor settings; its applicability extends outdoors at the natural location of cultural heritage assets. To this end, immersive reality technologies, interaction methods, development platforms and mapping and cloud storage services have been combined to realise the interaction method. The Walkable MxR Map allows users to interact with virtual objects via maps that are virtually projected on the floor and viewable through MxR devices, specifically the Microsoft HoloLens. The projected maps are room-scale and walkable with a potential global scalability. Besides movement-based interaction, users can interact with virtual objects, multimedia content and 3D models using HoloLens's standard gesture, gaze and voice interaction methods.

Introduction

The roles of immersive reality technologies, such as Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MxR), in terms of enabling engaging interaction with virtual content and enriching visiting experiences in museums and heritage sites, have been demonstrated in the past (Anthes et al., 2016; Bekele and Champion, 2019a, 2019b; Bekele et al., 2018). Similarly, VH studies and applications have highlighted the crucial roles that Human-Computer-Interaction (HCI) methods play in terms of enabling the dissemination and acquisition of cultural knowledge and significance from VH applications and digital systems implemented in museums and heritage sites (Addison and Gaiani, 2000; Adhani and Rambli, 2012; Anthes et al., 2016; Katifori et al., 2019).

Following the visible trend of immersive reality technology and HCI methods’ adoption in the VH domain, the role of these technologies to enhance cultural learning in VH applications is becoming an area of interest in the domain (Caputo et al., 2016; Ibrahim and Ali, 2018; Ibrahim et al., 2011; Maye et al., 2017; McGookin et al., 2019). A recent article that attempted to compare existing immersive reality technologies and interaction methods against their potential to enable collaboration, engagement and contextual relationship in VH applications identifies MxR and collaborative and multimodal interaction methods as ideal tools for VH applications that target cultural learning (Bekele and Champion, 2019a).

The design and implementation of the interaction interface proposed as ‘Walkable MxR Map’ in this paper will primarily revolve around establishing the base platform for enabling collaboration, engagement and contextual relationship in VH applications, while cultural learning is placed at the centre of the whole design and implementation process. Interested readers may find the following articles for detailed discussion of cultural leaning in virtual environment, collaboration, engagement and contextual relationship (Bekele and Champion, 2019a; Ibrahim and Ali, 2018; McGookin et al., 2019; Šašinka et al., 2019; Tost and Economou, 2009). Here, it is worth it briefly discussing what collaboration, engagement, contextual relationship and cultural learning are from VH perspective.

Collaboration refers to the capability of virtual environments and interaction methods to allow either a co-located or remote collaboration between two or more users of VH applications. Collaboration can be considered as both an aspect of VH experience and a form of interaction method. In both cases, the collaborative environment/method mimics or it reflects users' or visitors' experience as it would be at physical museums or heritage sites. Enabling collaboration requires more than a collaborative interaction with a virtual simulation/reconstruction of cultural heritage. It also requires the implemented VH application to influence users’ experiential aspects as a result of their collective actions.

Engagement relates to the ability of virtual environments and interaction methods to enable and facilitate engaging experiences as a result of a combination of spatial and contextual immersivity (3D virtual environments and meaningful content) and intuitive interaction with the cultural context in the virtual environment. To this end, VH applications rely on interaction methods, immersive headsets (VR and MxR devices), and relevant cultural context. For instance, combining a tangible interaction method with highly immersive virtual environment and a relevant cultural context can be as engaging as a physical visit in museums and heritage sites (Katifori et al., 2019). Hence, VH applications that balance cultural context, interaction and immersivity can lead to enhanced cultural learning.

Contextual relationship refers to establishing a contextual relationship between users, cultural context, and the immersive reality systems. Existing VH applications that adopt immersive reality technologies for cultural knowledge dissemination focus on users’ interaction with the VH applications (Caggianese et al., 2018; Brett Ridel et al., 2014a, Ridel et al., 2014b; Schaper et al., 2017; tom Dieck and Jung, 2017). However, in order for VH applications to enhance cultural learning, establishing a contextual relationship between users, their physical surroundings (museums and heritage sites), and the virtual environment (cultural content) is as crucial as enabling intuitive interaction with the virtual environment. Hence, the relationship factor can be further categorised into three: relationship between user and reality (User-Reality relationship), relationship between user and virtuality (User-Virtuality relationship), and relationship between reality and virtuality (Reality-Virtuality relationship). An ideal immersive reality scenario will combine these elements into a User-Reality-Virtuality (URV) relationship (Bekele and Champion, 2019b).

The central objective of the interaction interface ‘Walkable MxR Map’ is, therefore, to propose and implement a base platform that can be adopted by the VH studies that target at balancing/integrating collaboration, engagement and contextual relationship as crucial interaction design elements of VH applications. As such, this paper reports on the details of the design and implementation of the interface.

At a high-level view, the proposed interface is a map-based interaction method in a Mixed Reality (MxR) environment. The interaction method is implemented by combining immersive reality, mapping and cloud storage services, and immersive reality application development platforms. The interaction method enables users to interact with virtual content via interactive and walkable virtual maps projected on or around the user's immediate surrounding. The maps serve as interaction metaphor, gateway to linked digital records of artefacts, and stage for presenting virtual content such as 3D models and audio-visual multimedia content relevant to a specific cultural or historical context in a museum environment and heritage sites. Hence, geospatial information sources (mapping services), engagement, and interactivity are the crucial interaction design aspects/elements of the map-based interaction method. The contributions of the Walkable MxR Map to the VH domain are summarised as follows:

  • (A)

    The application enhances visiting experience at museums and heritage sites. Conventional museums and heritage sites don't allow physical manipulation of artefacts. In such cases, users acquire further information about the artefacts through printed media and digital multimedia content displayed on screens next to the artefacts. The Walkable MxR Map, however, will enable users to manipulate the digital representations of the artefacts (3D models) via interactive and immersive MxR visualisation environment.

  • (B)

    The approach will be an addition to the fairly new application category (theme) of the VH domain that attempts to adopt and disseminate immersive reality technologies for the promotion of cultural learning.

  • (C)

    The implementation of this approach contributes towards extending the existing expertise that tackles the technical challenges of combining geospatial information and immersive reality technologies across various domains.

  • (D)

    The proposed approach will serve as an initial platform for cultural learning themed VH application that attempt to combine cloud services, multiple geospatial and multimedia sources, immersive reality and interaction interfaces.

The remainder of the paper is organised as follows. Section 2 will discuss existing and related studies and VH applications. Section 3 System architecture and components of the walkable MxR map, 4 Implementation of the walkable MxR map as interaction interface for Virtual Heritage will provide detailed discussion on the design, system components and architecture of the proposed interaction interface and its prototypical implementation along with technological aspects and requirements. Finally, Section 5 Discussion, 6 Conclusions will discuss and conclude the results of the prototypical implementation, limitations of the approach and areas of improvement for future works to extend the applicability of the Walkable MxR Map.

Section snippets

Related works

Immersive reality technologies play an important role in cultural heritage and education (cultural learning). These technologies can enhance visitors/users' interest, attention and engagement in museums and heritage sites. For instance, De Paolis et al. (2018) leverage AR to help students understand spatial concepts and the working principles of complex mechanical components of machines designed by Leonardo Da Vinci. The authors use a mobile augmented reality application that shows the overall

System architecture and components of the walkable MxR map

The Walkable MxR Map has five major components: Head-Mounted-Display, Geospatial Information and Event Cue, Interaction Inputs and Mixed Reality (MxR) Framework, Event and Spatial Query Handler, and Cultural Dataset containing historical and cultural context (3D models, multimedia content and event spatiotemporal information). Fig. 3 shows the overall system architecture and its major components. A detailed discussion on the overall architecture and on each component is presented below. The

Implementation of the walkable MxR map as interaction interface for Virtual Heritage

The system architecture discussed above was implemented using a commercial wearable device (HMD), proprietary software, opensource toolkits, and custom-written scripts. This section will discuss these tools, technologies and limitations encountered during the implementation.

The discussion that focuses on the limitations associated with the adopted tools and technologies might be useful to readers interested in adopting the system architecture across domains and diverse application areas. As

Discussion

The implementation of the Walkable MxR Map was realised using the technologies and services discussed above. This section will provide discussion on the actual prototype, expected experiential aspects, and limitations encountered during the implementation.

Fig. 5 shows a virtual map projected on the floor along with Event Cues and 3D models. It is possible to interact with the virtual environment via gaze, gesture, voice, and movement. The map covers an area of two square meter. However, users

Conclusions

A novel map-based interaction interface, namely Walkable MxR Map, has been proposed and implemented in this paper. The central objective of the application is enhancing cultural learning in VH applications by enabling a contextual relationship and interaction between users, immersive reality technologies and cultural context. To this end, a system architecture consisting of five components has been proposed. In addition, a working prototype has been built using custom, proprietary and

Declaration of competing interest

I declare that there is no conflict of interest.

I Mafkereseb Kassahun Bekele declare that there is no conflict of interest and I am the sole author of this submission.

Acknowledgments

To Dr M. McCarthy, Dr Deb Shefi and Kevin Edwards from Western Australia Shipwrecks Museum, for the access to SS Xantho 3D models (used during the implementation of the Walkable MxR MaP) and historical collections.

References (38)

  • L. Bollini et al.

    Geolocalization as way finding and user experience support in cultural heritage locations

  • F.M. Caputo et al.

    Gestural interaction and navigation techniques for virtual museum experiences

  • J. Carmigniani et al.

    Augmented reality technologies, systems and applications

    Multimed. Tools Appl.

    (2011)
  • D. D'Auria et al.

    A 3D audio augmented reality system for a cultural heritage management and fruition

    J. Digit. Inf. Manag.

    (2015)
  • L.T. De Paolis et al.

    Augmented reality to understand the Leonardo's machines

  • A. Favre-Brun et al.

    Revealing the "spirit of the place": genius Loci, a spatial augmented reality performance based on 3D data and historical hypotheses

  • P. Fogliaroni

    Mixed reality for archeology and cultural heritage

  • P. Girão et al.

    Real-time multi-view grid map-based spatial representation for mixed reality applications

  • N. Ibrahim et al.

    A conceptual framework for designing virtual heritage environment for cultural learning

    J. Comput. Cult. Heri. (JOCCH)

    (2018)
  • Cited by (26)

    • Mixed reality experiences in museums: Exploring the impact of functional elements of the devices on visitors’ immersive experiences and post-experience behaviours

      2022, Information and Management
      Citation Excerpt :

      Museums and other cultural heritage organisations exploit MR opportunities to reshape physical space and heritage exhibitions (e.g., monuments, archaeological sites, paintings and historical artefacts). They capitalise on new technological multi-sensory environments and the value of digital storytelling (Loureiro et al., 2019), redefining the relationship between visitors and the tangible and intangible cultural heritage (Bekele, 2019; Flavián et al., 2019; Rahaman et al., 2019; Trunfio et al., 2022; Wang and Xia, 2019). Scholars tend to emphasise the disruptive role of new realities in the museum setting by reframing traditional cultural experiences, such as heritage preservation and promotion, and transforming the cultural site visit into a multi-experiential, sensorial, and emotional experience (Bec et al., 2019; Fenu and Pittarello, 2018; Flavián et al., 2021; Guttentag, 2010; Trunfio et al., 2020, 2022).

    • A theoretical framework for designing smart and ubiquitous learning environments for outdoor cultural heritage

      2020, Journal of Cultural Heritage
      Citation Excerpt :

      Given that, technologies for cultural heritage contexts need some extra aspects to be considered, which were not considered in the mentioned frameworks and models, such as: (a) the content that learners/visitors consume to perceive history; and (b) interaction with the contexts, which could involve some important aspects such as: activities that visitors perform to take learning opportunities, resources and tools that mediated the performance, information format, and, the interface design that learners use to access services and activities. Besides the aforementioned studies, some other were introduced for cultural spaces but not necessarily for outdoors, as mentioned earlier; they were not dedicated for introducing a framework for enhancing learning from sites [20–25,65,66]. However, non of them was for designing a smart environment to support learning on-the-move at outdoor cultural heritage sites.

    View all citing articles on Scopus
    View full text