Walkable Mixed Reality Map as interaction interface for Virtual Heritage
Introduction
The roles of immersive reality technologies, such as Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MxR), in terms of enabling engaging interaction with virtual content and enriching visiting experiences in museums and heritage sites, have been demonstrated in the past (Anthes et al., 2016; Bekele and Champion, 2019a, 2019b; Bekele et al., 2018). Similarly, VH studies and applications have highlighted the crucial roles that Human-Computer-Interaction (HCI) methods play in terms of enabling the dissemination and acquisition of cultural knowledge and significance from VH applications and digital systems implemented in museums and heritage sites (Addison and Gaiani, 2000; Adhani and Rambli, 2012; Anthes et al., 2016; Katifori et al., 2019).
Following the visible trend of immersive reality technology and HCI methods’ adoption in the VH domain, the role of these technologies to enhance cultural learning in VH applications is becoming an area of interest in the domain (Caputo et al., 2016; Ibrahim and Ali, 2018; Ibrahim et al., 2011; Maye et al., 2017; McGookin et al., 2019). A recent article that attempted to compare existing immersive reality technologies and interaction methods against their potential to enable collaboration, engagement and contextual relationship in VH applications identifies MxR and collaborative and multimodal interaction methods as ideal tools for VH applications that target cultural learning (Bekele and Champion, 2019a).
The design and implementation of the interaction interface proposed as ‘Walkable MxR Map’ in this paper will primarily revolve around establishing the base platform for enabling collaboration, engagement and contextual relationship in VH applications, while cultural learning is placed at the centre of the whole design and implementation process. Interested readers may find the following articles for detailed discussion of cultural leaning in virtual environment, collaboration, engagement and contextual relationship (Bekele and Champion, 2019a; Ibrahim and Ali, 2018; McGookin et al., 2019; Šašinka et al., 2019; Tost and Economou, 2009). Here, it is worth it briefly discussing what collaboration, engagement, contextual relationship and cultural learning are from VH perspective.
Collaboration refers to the capability of virtual environments and interaction methods to allow either a co-located or remote collaboration between two or more users of VH applications. Collaboration can be considered as both an aspect of VH experience and a form of interaction method. In both cases, the collaborative environment/method mimics or it reflects users' or visitors' experience as it would be at physical museums or heritage sites. Enabling collaboration requires more than a collaborative interaction with a virtual simulation/reconstruction of cultural heritage. It also requires the implemented VH application to influence users’ experiential aspects as a result of their collective actions.
Engagement relates to the ability of virtual environments and interaction methods to enable and facilitate engaging experiences as a result of a combination of spatial and contextual immersivity (3D virtual environments and meaningful content) and intuitive interaction with the cultural context in the virtual environment. To this end, VH applications rely on interaction methods, immersive headsets (VR and MxR devices), and relevant cultural context. For instance, combining a tangible interaction method with highly immersive virtual environment and a relevant cultural context can be as engaging as a physical visit in museums and heritage sites (Katifori et al., 2019). Hence, VH applications that balance cultural context, interaction and immersivity can lead to enhanced cultural learning.
Contextual relationship refers to establishing a contextual relationship between users, cultural context, and the immersive reality systems. Existing VH applications that adopt immersive reality technologies for cultural knowledge dissemination focus on users’ interaction with the VH applications (Caggianese et al., 2018; Brett Ridel et al., 2014a, Ridel et al., 2014b; Schaper et al., 2017; tom Dieck and Jung, 2017). However, in order for VH applications to enhance cultural learning, establishing a contextual relationship between users, their physical surroundings (museums and heritage sites), and the virtual environment (cultural content) is as crucial as enabling intuitive interaction with the virtual environment. Hence, the relationship factor can be further categorised into three: relationship between user and reality (User-Reality relationship), relationship between user and virtuality (User-Virtuality relationship), and relationship between reality and virtuality (Reality-Virtuality relationship). An ideal immersive reality scenario will combine these elements into a User-Reality-Virtuality (URV) relationship (Bekele and Champion, 2019b).
The central objective of the interaction interface ‘Walkable MxR Map’ is, therefore, to propose and implement a base platform that can be adopted by the VH studies that target at balancing/integrating collaboration, engagement and contextual relationship as crucial interaction design elements of VH applications. As such, this paper reports on the details of the design and implementation of the interface.
At a high-level view, the proposed interface is a map-based interaction method in a Mixed Reality (MxR) environment. The interaction method is implemented by combining immersive reality, mapping and cloud storage services, and immersive reality application development platforms. The interaction method enables users to interact with virtual content via interactive and walkable virtual maps projected on or around the user's immediate surrounding. The maps serve as interaction metaphor, gateway to linked digital records of artefacts, and stage for presenting virtual content such as 3D models and audio-visual multimedia content relevant to a specific cultural or historical context in a museum environment and heritage sites. Hence, geospatial information sources (mapping services), engagement, and interactivity are the crucial interaction design aspects/elements of the map-based interaction method. The contributions of the Walkable MxR Map to the VH domain are summarised as follows:
- (A)
The application enhances visiting experience at museums and heritage sites. Conventional museums and heritage sites don't allow physical manipulation of artefacts. In such cases, users acquire further information about the artefacts through printed media and digital multimedia content displayed on screens next to the artefacts. The Walkable MxR Map, however, will enable users to manipulate the digital representations of the artefacts (3D models) via interactive and immersive MxR visualisation environment.
- (B)
The approach will be an addition to the fairly new application category (theme) of the VH domain that attempts to adopt and disseminate immersive reality technologies for the promotion of cultural learning.
- (C)
The implementation of this approach contributes towards extending the existing expertise that tackles the technical challenges of combining geospatial information and immersive reality technologies across various domains.
- (D)
The proposed approach will serve as an initial platform for cultural learning themed VH application that attempt to combine cloud services, multiple geospatial and multimedia sources, immersive reality and interaction interfaces.
The remainder of the paper is organised as follows. Section 2 will discuss existing and related studies and VH applications. Section 3 System architecture and components of the walkable MxR map, 4 Implementation of the walkable MxR map as interaction interface for Virtual Heritage will provide detailed discussion on the design, system components and architecture of the proposed interaction interface and its prototypical implementation along with technological aspects and requirements. Finally, Section 5 Discussion, 6 Conclusions will discuss and conclude the results of the prototypical implementation, limitations of the approach and areas of improvement for future works to extend the applicability of the Walkable MxR Map.
Section snippets
Related works
Immersive reality technologies play an important role in cultural heritage and education (cultural learning). These technologies can enhance visitors/users' interest, attention and engagement in museums and heritage sites. For instance, De Paolis et al. (2018) leverage AR to help students understand spatial concepts and the working principles of complex mechanical components of machines designed by Leonardo Da Vinci. The authors use a mobile augmented reality application that shows the overall
System architecture and components of the walkable MxR map
The Walkable MxR Map has five major components: Head-Mounted-Display, Geospatial Information and Event Cue, Interaction Inputs and Mixed Reality (MxR) Framework, Event and Spatial Query Handler, and Cultural Dataset containing historical and cultural context (3D models, multimedia content and event spatiotemporal information). Fig. 3 shows the overall system architecture and its major components. A detailed discussion on the overall architecture and on each component is presented below. The
Implementation of the walkable MxR map as interaction interface for Virtual Heritage
The system architecture discussed above was implemented using a commercial wearable device (HMD), proprietary software, opensource toolkits, and custom-written scripts. This section will discuss these tools, technologies and limitations encountered during the implementation.
The discussion that focuses on the limitations associated with the adopted tools and technologies might be useful to readers interested in adopting the system architecture across domains and diverse application areas. As
Discussion
The implementation of the Walkable MxR Map was realised using the technologies and services discussed above. This section will provide discussion on the actual prototype, expected experiential aspects, and limitations encountered during the implementation.
Fig. 5 shows a virtual map projected on the floor along with Event Cues and 3D models. It is possible to interact with the virtual environment via gaze, gesture, voice, and movement. The map covers an area of two square meter. However, users
Conclusions
A novel map-based interaction interface, namely Walkable MxR Map, has been proposed and implemented in this paper. The central objective of the application is enhancing cultural learning in VH applications by enabling a contextual relationship and interaction between users, immersive reality technologies and cultural context. To this end, a system architecture consisting of five components has been proposed. In addition, a working prototype has been built using custom, proprietary and
Declaration of competing interest
I declare that there is no conflict of interest.
I Mafkereseb Kassahun Bekele declare that there is no conflict of interest and I am the sole author of this submission.
Acknowledgments
To Dr M. McCarthy, Dr Deb Shefi and Kevin Edwards from Western Australia Shipwrecks Museum, for the access to SS Xantho 3D models (used during the implementation of the Walkable MxR MaP) and historical collections.
References (38)
- et al.
Evaluation of spatial interaction techniques for virtual heritage applications: a case study of an interactive holographic projection
Future Gener. Comput. Syst.
(2018) - et al.
Investigating tangential access for location-based digital cultural heritage applications
Int. J. Hum. Comput. Stud.
(2019) - et al.
From photo to 3D to mixed reality: a complete workflow for cultural heritage visualisation and experience
Digit. Appl. Archaeol. Cult. Herit.
(2019) - et al.
Value of augmented reality at cultural heritage sites: a stakeholder approach
J. destin. Mark. Manag.
(2017) - et al.
Virtualized architectural heritage: new tools and techniques
IEEE MultiMedia
(2000) - et al.
A survey of mobile augmented reality applications
- et al.
State of the art of virtual reality technology
- et al.
A comparison of immersive realities and interaction methods: cultural learning in virtual heritage
Fron. Robo. AI
(2019) - et al.
Redefining mixed reality: user-reality-Virtuality and virtual heritage perspectives
- et al.
A survey of augmented, virtual, and mixed reality for cultural heritage
J. Comput. Cult. Heri. (JOCCH)
(2018)
Geolocalization as way finding and user experience support in cultural heritage locations
Gestural interaction and navigation techniques for virtual museum experiences
Augmented reality technologies, systems and applications
Multimed. Tools Appl.
A 3D audio augmented reality system for a cultural heritage management and fruition
J. Digit. Inf. Manag.
Augmented reality to understand the Leonardo's machines
Revealing the "spirit of the place": genius Loci, a spatial augmented reality performance based on 3D data and historical hypotheses
Mixed reality for archeology and cultural heritage
Real-time multi-view grid map-based spatial representation for mixed reality applications
A conceptual framework for designing virtual heritage environment for cultural learning
J. Comput. Cult. Heri. (JOCCH)
Cited by (26)
A framework study on the use of immersive XR technologies in the cultural heritage domain
2023, Journal of Cultural HeritageMixed reality experiences in museums: Exploring the impact of functional elements of the devices on visitors’ immersive experiences and post-experience behaviours
2022, Information and ManagementCitation Excerpt :Museums and other cultural heritage organisations exploit MR opportunities to reshape physical space and heritage exhibitions (e.g., monuments, archaeological sites, paintings and historical artefacts). They capitalise on new technological multi-sensory environments and the value of digital storytelling (Loureiro et al., 2019), redefining the relationship between visitors and the tangible and intangible cultural heritage (Bekele, 2019; Flavián et al., 2019; Rahaman et al., 2019; Trunfio et al., 2022; Wang and Xia, 2019). Scholars tend to emphasise the disruptive role of new realities in the museum setting by reframing traditional cultural experiences, such as heritage preservation and promotion, and transforming the cultural site visit into a multi-experiential, sensorial, and emotional experience (Bec et al., 2019; Fenu and Pittarello, 2018; Flavián et al., 2021; Guttentag, 2010; Trunfio et al., 2020, 2022).
A theoretical framework for designing smart and ubiquitous learning environments for outdoor cultural heritage
2020, Journal of Cultural HeritageCitation Excerpt :Given that, technologies for cultural heritage contexts need some extra aspects to be considered, which were not considered in the mentioned frameworks and models, such as: (a) the content that learners/visitors consume to perceive history; and (b) interaction with the contexts, which could involve some important aspects such as: activities that visitors perform to take learning opportunities, resources and tools that mediated the performance, information format, and, the interface design that learners use to access services and activities. Besides the aforementioned studies, some other were introduced for cultural spaces but not necessarily for outdoors, as mentioned earlier; they were not dedicated for introducing a framework for enhancing learning from sites [20–25,65,66]. However, non of them was for designing a smart environment to support learning on-the-move at outdoor cultural heritage sites.
Virtual and Augmented Experience in Virtual Learning Tours †
2023, Information (Switzerland)