Keywords

1 Introduction and Related Work

According to Greenbaum [9], “Virtual Reality is an alternate world filled with computer generated images that respond to human movements”. Virtual Reality (VR) is an interdisciplinary field of research, including the domains of human-computer interaction (HCI), computer-generated imagery (CGI) and computer graphics. Virtual reality technologies (VR) offer unprecedented user experience when it comes to 3D visualization [18], constituting a domain that is employed in a variety of fields.

VR systems make users perceive that they are physically present in the rendered non-physical world. This feeling of immersion is accomplished through the combination primarily of vision and secondarily of sound. Other approaches involve additional human senses such as touch [12] and olfaction [5]. Virtual environments (VEs) are used to let users interact with places, characters or objects, where they should experience the feeling of “being there”, also known as presence [2]. For various VR/MR/AR applications, such as virtual usability studies, it is fundamental that the participants have the feeling that they are really in the environment. Two important factors that influence presence are the level of immersion and the navigation method [14]. Jung’s et al. [13] research showed that social presence in mixed (VR & AR) environments is a strong predictor of four realms of experience, i.e. education, esthetics, escapism and entertainment. Immersion in VR environments can be helpful for further engaging people into the virtual world, suspending the feeling of disbelief and stepping out of the real world into another dimension through which they can accomplish their own goals. These goals can be set by the creators of the virtual environments so that they are constructive and helpful across a variety of domains.

In the healthcare domain, the advantages of the application of Virtual Reality are twofold: it can be applied either for the training of doctors in a realistic environment without endangering a person’s health (e.g. surgical skills training [8 and 19]) or for immersing patients in virtual environments and motivating them to perform rehabilitation exercises in a pleasant manner [4].

Tourism constitutes an additional area of VR application. Several implications of virtual reality are presented by Guttentag [11] in the domain of tourism, including planning and management, marketing, entertainment, education, accessibility and heritage preservation. Guerra et al. [10] claim that one of the promises of virtual reality is that when a user observes a monument, a building or a sculpture the perception of additional information will be straightforward and engaging.

VR is an extremely promising tool for the enhancement of learning, education, and training [16]. VR technologies are employed in collaborative virtual environments for the development of learning and communicating skills, such as E-Teatrix [15]. Furthermore, serious games are employed for telling stories and narrations for educational purposes [21], allowing users to experience a personalized view of mythology.

Cultural Heritage is an information-rich domain which is fruitful for visualization. Limitations in exploring institutions such as museums are obstacles often experienced by visitors [20]. Another issue when exploring public institutions is time constraints, which may force users to explore information at a higher speed than preferred [7]. In this direction, visualizations of virtual environments regarding cultural heritage can enrich and supplement, but not replace, in vivo visits to institutions. Sooai et al. [17] created virtual 3D environments presenting 3D models which users can experience by employing virtual reality views using mobile phones.

Furthermore, monuments can be inaccessible for a variety of reasons: the excavation process is not yet complete, renovations are taking place or even their state is fragile. Therefore, visualizations of such sites can provide access to people who would not be otherwise able to explore them. Additionally, items may be located in a protected area like a museum. In this case, the only way to perceive an environment containing the various elements found inside it is by augmenting existing environments with the items or by incorporating virtual environments containing the items.

Barsanti et al. [3] concluded that museums are among the first organizations to make use of advanced VR technologies to investigate their educational potential determining how they provide public education and amusement. The combination of information and images that can be interactively manipulated by the user and the immersive experience are two basic advantages of VR technologies: the user can both experience the sensation of being in an ancient world and can actively participate in the virtual environment. Finally, a 3D cave display enriched with haptic interaction modalities in order to recreate the process of an ancient monument construction is presented in [6], combining virtual reality with playful interaction.

This paper discusses ongoing work regarding immersive environments in which users can interact with cultural heritage 3D objects, navigate in a virtual reality environment and be informed using their bare hands. The proposed framework can be easily adapted to different domains, such as education or healthcare, keeping the information visualization components and interaction techniques the same, thus creating an immersive and interactive framework for VR environments.

2 Interacting with Cultural Heritage 3D Objects in VR

The visual representation of objects can be in the form of a three dimensional model, reconstructing digitally its physical shape in high detail. Digital 3D models can be utilized when the item itself is not available to exhibit, or when the item is too small or large scale, making it impractical to examine it in detail. In these cases, the exploration of an item can be performed through an interactive visualization of its reconstruction.

The system allows users to grasp three dimensional items and manipulate them, i.e., move, rotate or scale them in virtual space (Fig. 1). The application employs for each exhibit a metallic item (manipulator) which the users can grasp with one hand to move and rotate. The manipulator only appears if the user’s hand is near the item. Once holding an item from the manipulator, another metallic item appears below the item with a sphere on the other side, which the users can grab with their other hand in order to scale (maximize or minimize) the exhibit.

Fig. 1.
figure 1

Manipulation of a 3D cultural heritage object in a VR environment

Full size image

The users’ left hand is used as a menu which is shown when the palm is facing the users’ eyes. This menu refers to the last item selected and can be pinned at any time using the pin button at the top right side. It contains an indicative title and image, along with a short textual description of the item. Furthermore, actions on the corresponding exhibit can be applied through the options at the bottom side of the menu. These currently include lighting, auto rotation and position reset. Upon lighting selection two draggable spotlights are enabled which users can place towards the exhibit in order to reveal more details of the item or to focus on a specific area. Users can disable the lighting whenever they want to. Auto rotating the exhibit is available so the users can position the object wherever they want, in their preferable scale and lighting and enable the auto rotation along Y axis in order to see a panoramic view of it. Finally, they can reset the exhibit to the initial position-scale and lighting.

The system can be employed in various domains, including:

  • Education: it can be used for educating children on anatomy and biology, engineers by breaking down complex items into components, etc.

  • Cultural Heritage: for items which are unavailable to citizens

  • Healthcare: it can be used for rehabilitation (e.g. stroke) and fine motor skills improvement

  • Professional environments: it can be used for employee training, like a simulation or a complex procedure

  • Advertisement: preview and promotion of the advertised product

In general, it constitutes a case study of immersive visualizations using a VR headset and natural interaction through gestures, involving a one to one mapping to the physical world. The hardware used includes:

  • Oculus Rift: A head-mounted display used for stereoscopic rendering, providing a virtual reality experience.

  • LeapMotion controller camera, allowing the tracking of user’s hands and providing tracking information about hand articulations, and therefore allowing the rendering of corresponding hands and the recognition of gestures.

2.1 Interaction

The interaction techniques for VR environment presented in this paper are based on the Leap Motion sensor placed in front of an Oculus Rift (Fig. 2), which displays a virtual world to the users. This setup allows free user movement in space, enabling them to turn their head towards any direction. Gesture recognition is accomplished with the camera placed in front of the user’s head and therefore the user’s hands are never occluded by the user’s torso, which is a shortcoming for different setups where the depth sensor is placed in a static position. Egocentric manipulation techniques, such as the virtual hand, translate the user’s hand movements to a simplified virtual representation of the hand, in which objects are typically glued to the virtual hand upon contact [1]. Virtual hand metaphors can be enhanced by providing an increased control of the virtual hand (e.g. finger motions) and providing additional visual feedback.

Fig. 2.
figure 2

User interacting with a virtual 3D statue

Full size image

The developed system includes a fully animated virtual hand (including the forearm) which follows the user’s arm, hand and finger movements. The 3D model was directly obtained from the Leap Motion SDK. Hand tracking was provided by the SDK, allowing to track the forearm, the hand and the fingers of the user’s dominant hand. It was decided to use Leap Motion because it provides seamless finger tracking without the need to wear gloves or markers, thus providing unobtrusive tracking. The tracking quality ensures a correct animation for the realistic virtual hand for most situations, still, when the palm is in a vertical position finger tracking issues appear due to the palm inter-finger occlusions. The selection is achieved by a virtual “click in the air” with the pointer finger. The open left palm reveals the information menu accompanied with extra functionality which can be applied to the 3D object, such as lighting, position and rotation change. Finally, in order to select and move objects, a pinch with two fingers is sufficient, and upon release of the pinch the object is also released from grabbing.

3 Conclusion and Future Work

This paper has presented ongoing work regarding immersive and interactive environments in which users can interact with cultural heritage 3D objects using their hands and body movement, and therefore enhancing user experience in cultural heritage information visualization. The ultimate objective of this work is to explore the potential of creating a framework with which, regardless of the context of use, users will interact in virtual reality environments, manipulate 3D objects using only their hands/body without wearable devices and will perceive information through a common information visualization template. The system was temporarily installed in several public spaces where users had the opportunity to interact and provide feedback regarding their experience. The early users’ comments on applying gestural interaction in combination with VR devices were very encouraging, as the approach proved to be natural, usable and entertaining. The next planned steps involve conducting an extensive evaluation, assessing the users’ preferences both among the proposed alternate gestural approaches and in comparison to more traditional devices.