Creating Room-Scale Interactive Mixed-Reality Worlds Using Off-the-Shelf Technologies

Abstract

Mixed-Reality (MR) represents a combination of real and virtual worlds. Off-the-shelf solutions already exist for quite some time which enable the creation of MR worlds where physical and digital objects co-exist and interact in real time. However, little is still known regarding methodological and technical challenges involved in the development of room scale MR worlds with the use of high-end technologies, especially when considering real-object representation. This paper reports hands-on experiences in creating a room-scale MR world, using widespread, off-the-shelf object-tracking technologies and 3D modelling techniques, which enable free user movement, accurate real-object representation in the virtual world, as well as interactivity between real and virtual objects.

1 Introduction

Virtual Reality (VR) refers to interactive synthetic worlds which often mimic the properties of real-world environments and allow user immersion. Notwithstanding, these synthetic worlds may exceed the limitations of physical reality, since any law, which naturally governs space, time, mechanics, material properties and so on, can be tempered with [1]. A related field of immersive technologies, which involves mixing of real and virtual elements, is referred to as Mixed Reality (MR) [1]. Intensive research has allowed the implementation of MR in fields such us gaming [2], education [3], industrial design [4] and medicine [5].

Recent advances in VR-related technology generated affordable and compact systems, such as Head Mounted Displays (HMDs), resulting to the commercial breach of VR systems since 2015. These developments have triggered various VR and MR expansions, especially by independent developers and small laboratories. According to ongoing research reports, it appears that the use of real objects in MR worlds, has a positive impact on users [6]. However, studies specifically addressing real-object usage in MR are scarce, whereas the techniques used to incorporate real objects in virtual worlds do not take advantage of recent technological standards, which greatly increase user comfort and quality of experience.

In this paper, we showcase the design and development of a room-scale MR environment which will provide a testbed for future experiments. More importantly, several critical aspects related to MR implementation are addressed such as free user movement, accurate real-to-virtual object representation in the VR environment, as well as interactivity between real and virtual objects. Therefore, this study supports future developers by providing a detailed report of the steps involved in creating a room-scale MR world with the use of widespread, off-the-shelf technologies. More specifically, in this paper we showcase the development process of a dungeon- themed MR world which incorporates:

• registration of user movement, performed in the real world, into the VR environment,

• accurate representation of texturally rich real objects (i.e. a rock and a wooden stick) in the VR environment,

• registration of accurate user-interaction with real objects into the VR environment, and

• interactivity between real and virtual objects.

The remainder of the paper is structured as follows: Sect. 2 discusses some seminal conceptual issues regarding VR and MR, as well as examples of MR applications; Sect. 3 presents the process followed to generate accurate virtual representations of real objects; Sect. 4 presents the process followed to materialize the MR world along with a number of emerging issues and finally, Sect. 5 concludes our work and draws directions for future research.

2 Related Research

The evident, ascending development growth of VR technologies is in accordance with what has been foretold, even though the pursuit of the “ultimate display” [7] is nowhere near its finish line. In 1995, Slater and Usoh refer to a totality of inputs and the need of total immersion for each sense separately [8] which could only mean the seamless expression of proprioceptive mechanisms, both conscious and non-conscious, to exteroceptive stimuli that at no point disrupt the otherwise naturally occurring environmental and bodily events. This understanding does not equal virtual worlds grounded in realism, as this is not a matter of content, but rather the taming of those realistic elements which would allow the introduction of any implausible content imaginable.

What precedes the ideal immersive virtual experience is a long list of conceptual and technical prerequisites. And, as complicated as those may be, intertwining the real with the virtual, particularly on the right-end side of the reality-virtuality (RV) continuum [9], is an additional, head-on challenge of whichever degree of immersion might have been achieved till that point. The human body and its sensorimotor channels, as the “primordial communication medium”, are indeed the perpetual central question [10] and the gravitational pull of an orbital quest. In other words, the simultaneous engagement of virtual and real events, elements and, more importantly for the present study, objects in MR worlds, deals with challenges on two fronts and by default, presents higher technological challenges as well [11].

Examples of MR, specifically Augmented Reality (AR) applications, can be traced back to Ivan Sutherland and the emergence of prototype, see-through displays projecting 3D graphics [11]. To date, several MR systems have been designed, providing users with high quality immersive experiences [12, 13].

For instance, one of such interactive MR systems is the eXperience Induction Machine (XIM), which is mostly used in psychology and human-artifact interaction related experiments. XIM consists of a physical room equipped with peripherally placed projector screens, a luminous floor, interactive sound systems, and several sensors (e.g. pressure sensors, microphones, cameras), allowing multi-user interaction, with both physical and virtual worlds, in real-time [14].

Another example is MR MOUT (Mixed Reality Military Operations in Urban Terrain), providing training simulation with the use of a video see-through HMD and re-creation of urban landscape. Using a combination of bluescreen technology and occlusion models, MR MOUT allows the trainees to move around a courtyard and hide behind objects, whilst real and virtual players emerge from portals and engage in close-combat battle [15].

Touch-Space is an ubiquitous, tangible, and social MR game-space, which, whilst using the physical and social aspects of traditional game-play, incorporates the real-world environment as an intrinsic and essential game element [16]. Touch-Space provides a full spectrum of game interaction as well as an embodied experience that starts off in the real, physical surroundings (human to human and human to physical world interaction) and gradually transcends through augmented reality (AR) towards the virtual environment [16].

Several other MR systems, developed by widespread technology providers, blend the real world with virtual ones. The accelerating emergence of such systems is indicative of the field’s industrial and marketing appeal. Real Virtuality¹ for instance, is a multi-user immersive platform, combining a 3D environment which can be seen and heard through a VR headset, with a real-life stage set. Users are tracked by a motion capture system, whilst being able to see their own bodies, move physically in the virtual environment and interact with physical objects. Whereas, VIRTSIM² is a highly immersive virtual reality system which can pit squads of players against each other in a realistic virtual reality combat environment. The system utilizes full-motion body capture, VR headsets, gun props, and even shock-feedback.

Even though MR systems like Real Virtuality and VIRTISM are not yet commercially available, room-scale VR experiences are; Oculus room-scale solution³ allows users to freely move in a room with the use of Oculus Rift.

Research and development of MR systems, that allow simultaneous interaction with a real object and its virtual representation, a process termed as passive haptics, are already being carried out [17]. Such a case is the creation of a virtual world on top of the physical one with the use of Tango enabled smartphone devices whilst users’ full body locomotion is tracked with Microsoft Kinect sensors allowing free movement. In this MR system real objects and their virtual representations differ in appearance, yet maintain some tactile characteristics (e.g. size and weight), to secure coherence in some sensory feedback. For example, the system allows users to touch two real, paper boxes, assigned with markers, in the physical world and interact with their virtual representations in the virtual world. Again however, boxes are easy to recreate in 3D and the system does not offer solutions for complex objects in regards of texture and morphology.

The technological advances presented above reveal a promising future for MR, where users will be able to freely move around real and virtual spaces simultaneously, while interacting with both real and virtual objects. However, the development of room-scale MR worlds like Real Virtuality and VIRTISM is a challenging process, especially since there are no guidelines in regards of technical setup, cost-effective solutions, or suggested methods of development. In addition, the aforementioned MR systems do not in fact include texturally rich objects and thus, there are no paradigms of complex real-object representations into MR worlds. As already mentioned, when users interact with a real-object’s virtual representation, tactile augmentation through touch and grasp increases the realism of the virtual environment in MR systems [6]. Thus, the ability to incorporate any object, regardless of texture and morphology, is critical for MR development.

3 Accurate Real-Object Representations

When designing an interactive MR world, one of the most crucial components is accurate real-to-virtual object representation. Dimensions of the virtual object should of course be identical to the original, but also, its virtual ‘physiology’, simulated via appropriate textures and materials, should also be in accordance with the original object. Immersed users who are meant to touch a real object, virtually represented in the VR environment, should not experience inconsistencies regarding tactile-to-visual correlating information.

Therefore, realistic correlation of characteristics is a requirement which can be met via conventional 3D modelling methods (e.g. polygonal modelling) only for simple objects, in regards of geometry and texture. Otherwise, the modelling process is strenuous and time consuming, especially for complex objects like the rock used for the present development of a MR experience. To address this issue, the 3D modelling techniques of Photogrammetry and 3D object Scanning were used, using low-cost methods and equipment.

3.1 Photogrammetry

Photogrammetry, with the use of image measurement and interpretation, delivers the form and location of an object by using one or more photographs of that object [18]. Nowadays, various accessible software solutions simplify the process of creating 3D models via photogrammetry. In this work Autodesk Remake⁴ was used.

In order to create an interactive 3D object with the use of digital imaging, a sequence of photographs is required, circling the perimeter of the object and alternating the capture angle at about 5° to 10° between each shot so that the resulting photographic sequence has approximately 70% of overlapping information [19]. Moreover, all photographs need to be sharp and overall capture all possible angles of the object. Finally, the room/space, where the Photogrammetry process is taking place, needs to have proper lighting conditions to achieve diffused illumination of the object, whilst high contrast, intense shadows and the use of a flash light are prohibited.

For the purposes of the present study the 3D model of a rock was generated using photogrammetry, to be later incorporated in a MR experience within a dungeon themed VR environment. Figure 1a shows the preparation of the room where the sequential photographing of the rock took place. The rock was elevated with the use of a tripod to expose the maximum number of visible angles, whereas artificial lighting ensured diffused lighting conditions.

Fig. 1.

(a) Photoshoot preparation (b) Initial photogrammetry results; (c) Final 3D model; (d) Photos positions.

The resulting sequence of photographs was imported into Autodesk Remake and the primitive 3D model was created. However, 3D objects, modelled via photogrammetry, often require additional tweaking, since surrounding objects may interfere with the extraction of object information through digital imaging (see Fig. 1b). Autodesk Remake offers a set of tools which allow alterations of the object and removal of excess information. The final 3D model representing the rock object is presented in Fig. 1c, while Fig. 1d shows the positions wherefrom the photos of the object were taken.

3.2 3D Scanning

The second method utilized to create 3D models for the purposes of the present study, was the 3D scanning of real objects, with the use of relevant technologies, such as structured light or laser scanners, to create detailed 3D models of real objects [20]. To perform 3D scanning in this work, the low-cost, 3D depth camera-enabled, Microsoft Kinect sensor was used. The Microsoft Kinect sensor is based on an infrared light (IR) projector and an IR camera which allows the capture of changes on an IR pattern (see Fig. 2a). The Microsoft Kinect sensor utilizes the abovementioned process to produce depth images (see Fig. 2b). Finally, an RGB camera allows the capture of color [21].

Fig. 2.

(a) Microsoft Kinect sensor IR image; (b) Microsoft Kinect Sensor depth image; (c) 3D Scanning of object using Microsoft Kinect sensor; (d) 3D Scanning result (3D model on the left) and texture replacement into Cinema 4D (3D model on the right).

Having installed the Microsoft Kinect sensor using the Kinect SDK⁵, the Skanect⁶ software has been used to 3D scan objects. The object chosen to be 3D scanned was a wooden stick, also placed on a tripod to ease the scanning process. After feeding the dimensions of the object into Skanect, the 3D scanning process began, moving the Microsoft Kinect sensor around the wooden stick to capture as much geometry as possible (see Fig. 2c). Then, the colorize function of the Skanect software was used to analyze data captured by the RGB camera of the Microsoft Kinect sensor, creating a texture to be applied on the 3D model.

Finally, the 3D model was exported from Skanect to the 3D graphics creation software, Cinema 4D⁷, where the additional unwanted geometry caused by surrounding objects was removed, finally producing the desired 3D model. However, the colorize function of Skanect produced low quality textures. Therefore, the wooden stick texture was replaced with one of higher quality, also representing a wood interface, created in Cinema 4D (see Fig. 2d).

It should be noted that, based on the software available to this study, the 3D models produced with the process of photogrammetry were better suited for a MR experience, in terms of texture quality and lower number of polygons. However, using photogrammetry to create 3D models of objects with slim parts, such as the wooden stick, resulted in inaccurate models, thus dictating 3D scanning as a preferred option.

Therefore, the choice of modelling method depends on various parameters and combination of software is mandatory for accurate real-to-virtual representation of objects, especially for MR worlds.

4 MR World Materialization

4.1 Real Object Registration to the MR World

Vicon’s motion capture system, along with its complementary software, Blade, were the crucial components of the present study⁸. The Vicon motion capture system is built upon infrared cameras (see Fig. 3a), commonly placed along the perimeter of a room’s ceiling, allowing the tracking of retroreflective markers in real space (see Fig. 3b).

Fig. 3.

(a) Infrared camera; (b) Retroreflective marker; (c) Retroreflective markers assignment; (d) Prop creation in Vicon Blade.

Apart from full-body human motion capture, which is Vicon’s main purpose, tracking of prop objects is also possible by placing retroreflective markers around an object. To be able to track the position of the rock and the wooden stick objects in the real space, both were assigned with at least five (5) retroreflective markers (Fig. 3c) to ensure smooth tracking. Moreover, retroreflective markers were also assigned to an Oculus Rift Head Mounted Display (HMD), which was used by the user to navigate and experience the MR world (Fig. 3c).

Following the marker assignment, the objects were placed on the floor and a data sample, including location and rotation of markers, was recorded using Blade. After reconstructing the data sample, the retroreflective markers’ positions appear in Blade, wherefrom they can be selected in groups. Finally, each retroreflective markers-group can be converted into a prop object, by assigning a prop name and then calibrating the prop using Blade’s built-in prop creation tool (see Fig. 3d).

The process described above allows the real-time registration (i.e. solving) of the position and rotation of the real objects to the props through Blade (see Fig. 4a and b). It must be noted that Blade calculates the position of a prop using a bone object, with the first marker selected during prop creation serving as the root of the bone and the second marker selected to represent the end-point of the bone. Figure 4a, presents the rock item assigned with markers, while Fig. 4b shows the root, end-point and bone of the respective prop in Blade.

Fig. 4.

(a) Objects location in the real world; (b) Solving of real object location to the props in Blade.

Next, to successfully materialize the MR world created in this work, the Unity 3D engine has been used⁹. Unity 3D allows the creation of 3D interactive virtual worlds, whilst also allowing incorporation of Oculus Rift supported VR applications development. Firstly, a VR space representing a dungeon cell has been created, having the same square dimensions as the real room where the user was whilst experiencing the MR world. Then, the 3D objects, created using photogrammetry and 3D scanning, have been imported in the VR space (see Fig. 5a). 3D objects are often affected by the software used when creating or processing them. Therefore, once they are imported into Unity 3D, their scale factor may require adjustment to secure proper dimensions in relation to the real objects.

Fig. 5.

(a) Objects location in the virtual world; (b) Objects location in the real world.

The next step refers to the data flow between Blade and Unity 3D; a process which is typically handled by Vicon’s Tracker¹⁰ or the Pegasus¹¹ software. Since those however were unavailable, a cost-effective solution was implemented using Vicon’s Datastream SDK¹², along with a custom script created for Unity 3D. The custom script was based on the script used to enable interaction between Unity 3D and Vicon Pegasus¹³. More specifically, the script collected information sent through Blade’s data stream function and used the names of the props to match the 3D objects’ and real objects’ location and rotation¹⁴.

The Oculus Rift Core Package for Unity 3D¹⁵ allowed preparation and application of the Oculus Rift HMD in the MR world. This package includes tools which enable the integration of Oculus Rift HMD as part of a Unity 3D project, using simple drag and drop functions. Also, the Oculus Rift HMD comes with a separate sensor which tracks the HMD location up to a certain distance. However, the tracking process only works when the HMD is directly in front of the sensor, thus limiting the user’s range of movement whilst in the MR world. In addition, whilst trying to track the Oculus Rift prop location and rotation, the Oculus sensor interfered with the ones provided by Vicon, and therefore had to be disabled. Even though Oculus Rift comes with official instructions on how to implement a room scale solution, it is not suitable for real-time object tracking.

The processes described above results to an MR world wherein the user, in real-time, is free to move around, interact with real objects and their accurate virtual representations, and apply interaction/collision between real and virtual objects¹⁶.

4.2 Real-Time Hand Representation

Even though the above process was successful, several issues emerged when trying to implement hand interaction and subsequently test the accuracy of users’ touch with the real objects. First, Leap Motion¹⁷, an infrared sensor allowing precise hand tracking into VR environments was tested, to provide users with real-time hands visualization. However, the retroreflective markers placed on the real objects considerably affect Leap Motion’s performance. Figure 6a illustrates the issue while using Leap Motion as the left hand of the user (while holding the rock object) is not accurately tracked in VR¹⁸. Therefore, the use of Leap Motion is not recommended. This issue can be dealt with the use of alternative solutions for finger and hand movement tracking, which do not use infrared light, such as the ManusVR data glove¹⁹.

Fig. 6.

(a) Leap motion hand tracking and occlusion issue.

4.3 Object Tracking Accuracy Issues

When developing MR worlds with technologies such as the ones proposed in this paper, another issue to consider is proper adjustment of the 3D object’s pivot points (the center of the object’s rotational system), in order to enable accurate representation of real objects in the virtual world.

The pivot point of every 3D object should be adjusted at the same location as its respective prop’s root point, to increase the tracking precision and also avoid incidents such as rotation and location offsets during user interaction with the real objects.

To test the precision of this process we designed (using Cinema 4D) and 3D Printed a 10 × 10 cube, assigned it with retroreflective markers and finally registered it as a prop in Blade (see Fig. 7a and b). Moreover, the pivot point of the 3D model of the cube was assigned to the root of the prop item (see Fig. 7b and c). Finally, a coded script which acquired the markers positions from Blade and visualized them into Unity 3D, thus testing in detail the prop markers’ location both in real and virtual world (see Fig. 7d). This way we could accurately move and rotate the cube object²⁰ (see Fig. 7e).

Fig. 7.

(a) 3D Printed cube assigned with retroreflective markers; (b) 3D Printed cube prop in Blade; (c) Pivot point adjustment; (d) Markers visualization in Unity 3D; (e) 3D Printed cube registration in VR; (f) Accurate hand interaction with the 3D Printed cube in VR; (g) Accurate hand interaction with the rock object in VR.

To test accuracy of user touch on real objects and their virtual representations, we used a 3D hand model which represented a user’s real hand tracked by Blade in the virtual world (see Fig. 7f). The 3D hand had the same dimensions as the user’s hand. Tests examining interactivity of the virtual hand with both the cube²¹ and the rock²² object showed successful levels of accuracy, since the user was able to touch each detail of the objects, both in real and virtual world.

5 Conclusion and Future Work

This paper presented the process followed and challenges faced whilst creating an interactive MR world using off-the-shelf technologies. The MR world generated for the purposes of this paper, along with the implementation process, will serve as a testbed for further experimentation. However, the development process of the MR world, which can assist future developers, revealed useful directions such as:

• photogrammetry and 3D Scanning can be used to accurately represent real objects in the virtual world,

• object tracking accuracy is highly affected by the tracking method adopted; therefore, the pivot points and object scaling should be carefully treated as the tiniest offsets may greatly affect user experience,

• hand tracking accuracy is important and Leap Motion is incompatible with systems using retroreflective markers, therefore other solutions, such as ManusVR gloves, should be used.

Our future research plans involve the development of a prototype using Arduino²³ and Flex Sensors²⁴, which will allow finger interaction with virtual objects. We also intend to investigate the impact of real object usage (also incorporating 3D printed objects) as part of a MR experience and study its potential effect on users’ co-ordination and perceived level of immersion.