1 Introduction

The introduction of the Microsoft HoloLens [1], a commercial off-the-shelf augmented reality device, has allowed researchers at the United States Army Research Lab to explore using augmented reality technology for data visualization. The ability to superimpose data generated from a physics-based modeling and simulation into the actual physical environment is a very effective way of showing the results of the simulation. In addition, it is more than likely that a data visualization session is a collaboration between a group of researchers and stakeholders of the project. Shared augmented reality capability would be needed to support simultaneous visualization of the same 3D holographic object by multiple participants. Billinghurst et al. in 2000 explored future computing environments research using augmented reality as an interface for collaborative computing [2]. Microsoft HoloLens may be the missing hardware that is needed to make shared augmented reality workspace a reality.

Figure 1 illustrates the view from a Microsoft HoloLens for a mission planning shared augmented reality application where multiple users of Microsoft HoloLens collaborated on a mission planning scenario. Using the shared augmented reality application, users are able to collaborate on a mission plan by manipulating the same 3D holographic object of the aerial map displayed on their respective augmented reality display device while going through the mission objectives. Reducing registration error to ensure proper 3D holographic object placement in such a scenario is crucial to ensure all participants’ views are properly synchronized.

Fig. 1.
figure 1

Using augmented reality for scenario planning. A view from one of the Microsoft HoloLens showing landing and extraction route.

Full size image

Our experience with existing shared augmented reality demo application using HoloToolkit Sharing Service provided by Microsoft has indicated a noticeable differential in the 3D holographic object placement when the same object is viewed from different Microsoft HoloLens users. 3D holographic object registration is only accurate for the Microsoft HoloLens user doing the placement. For other Microsoft HoloLens users, the same 3D holographic object seemed to be slightly misplaced in the actual environment, resulting in less than ideal shared augmented reality experience. However, since Microsoft HoloLens is mainly a single user augmented reality device, a more accurately placed 3D holographic object for a shared augmented reality application may require additional 3D positioning information that is not available from the array of sensors incorporated into existing Microsoft HoloLens. Our framework used the OptiTrack motion capture system to add positional tracking data of the different Microsoft HoloLens users to improve the 3D holographic object registration in a shared augmented reality application. The main contribution of this paper is a Unity based sensor data fusion framework that reduces 3D holographic object registration error for a shared augmented reality application and demonstrated the application of the framework using our shared augmented reality mission planning scenario application.

In the remainder of the paper, the related work section discusses the significance of 3D holographic object registration for augmented reality technology. Section 3 describes our sensor data fusion framework and the comparative study design and results showing an improved 3D holographic object registration using our sensor data fusion framework. Then we discuss our shared augmented reality application implementation for a collaborative mission planning scenario in Sect. 4, and we conclude in Sect. 5.

2 Related Work

3D holographic object registration is one of the main challenges of the augmented reality technology. Azuma in his 1997 augmented reality survey paper devoted a whole section to registration challenges [3]. Zhou’s et al. survey paper published in 2008 on the trends in augmented reality research also mentioned a significant number of publications in augmented reality are related to registration research [4]. Zhou’s et al. paper also has a section on the use of hybrid tracking technique to improve overall tracking and 3D holographic registration accuracy for augmented reality applications.

Hybrid tracking technique combines tracking data from various 3D tracking systems in order to improve the overall tracking quality. Arth used sensors on a modern smart phone to improve 6-degree-of-freedom localizations in wide-area environments [5]. State used landmark tracking and magnetic tracking to provide superior augmented reality registration [6]. You used a hybrid of inertial and vision tracking to improve registration for an augmented reality application [7]. Although Zhou’s survey paper mentioned a significant portion of augmented reality research devoted to 3D holographic object registration, there are not a lot of publications targeting registration accuracy research for a shared augmented reality experience.

3 Sensor Data Fusion Framework

In our research to enable collaborative visualization, we addressed the issue of registration inaccuracy of 3D holographic objects in a shared augmented reality environment running on Microsoft HoloLens devices. In registering 3D holographic objects in the real world, Microsoft HoloLens uses their camera sensor inputs to process and generate coordinate systems needed for 3D holographic object placement. For a single user augmented reality experience, the object registration inaccuracy will only affect the view of the single user and there is no requirement to synchronize the views of all users which is required for a shared augmented reality experience. For a shared augmented reality experience with multiple devices, the goal is to minimize the registration error of the 3D holographic object so that all devices see an identical augmented reality world. Our sensor data fusion framework proposes a solution to minimize registration error by fusing external sensor data with the Microsoft HoloLens device’s sensor data to improve the registration accuracy for the multiple devices running a shared augmented reality application.

Figure 2 shows the development setup of our sensor data fusion framework. In our implementation, we used Unity to develop a client-server application, with the server being hosted on a dedicated server machine and the client applications deployed and run on the individual Microsoft HoloLens devices. We used three Microsoft HoloLens devices in our shared augmented reality application to demonstrate a synchronized view from three separate devices. For the external sensor data, we used OptiTrack Motion Capture system that uses the Motive software to convert the data captured by the 6 Infra-Red cameras setup we have into 3D positional and orientation data and broadcast the data over the network. The additional 3D positional and orientation data essentially provides the individual Microsoft HoloLens with its location information within a global coordinate system.

Fig. 2.
figure 2

Our shared augmented reality setup showing relative position of user A, user B, and user C (from left to right) in the real world looking at a 3D holographic object placed on top of the blue object in the real world. (Color figure online)

Full size image

We imported the global coordinate information into our Unity application using Unity-OptiTrack plugin and combined the data with the 3D positional data from Microsoft HoloLens sensors in our data fusion framework. Since OptiTrack uses marker-based tracking system, we attached markers with unique configuration to each Microsoft HoloLens for Motive to distinguish between the different Microsoft HoloLens devices.

In Unity, we used the 3D positional and orientation data of the Microsoft HoloLens streamed directly from Motive in our server application and fuse it with the sensor data of its corresponding Microsoft HoloLens device running the client application to reduce the registration errors. Having a dedicated server to host the application, the server has the exact information on the physical environment containing the positional and orientation data of each Microsoft HoloLens device and where the holographic object should be rendered with respect to the real world. Thus, each Microsoft HoloLens has a more concise understanding of its position in the global coordinate system and can register the 3D holographic object more accurately with this information. In other words, the server has an exact representation of the world and the location of the holographic objects, and the client application running on Microsoft HoloLens devices merely has to render the scene as defined by the server.

Although we used OptiTrack tracking system to stream the positional and orientation data, the application itself is not dependent on these specific components. Depending on the available sensors and how the data is captured, our data sensor fusion framework can be adapted to incorporate many different types of sensors to further improve tracking accuracy and 3D holographic objects registration. Using the sensor data processed by the data sensor fusion framework, our shared augmented reality application will be able to take advantage of multiple external sensors of different types to improve the object registration inaccuracy.

3.1 Implementation Results and Discussion

Initial use of our shared augmented reality application showed a vast improvement in the accuracy of the holographic object placement for all users compared to the holographic object placement in the shared augmented reality application based on Microsoft HoloToolkit Sharing Service. Computational scientists using the shared augmented reality application can now meaningfully collaborate on a visualization of the 3D holographic object like any objects in the physical world visible from the augmented reality device. When discussing certain characteristics of the 3D holographic object, scientists will be confident that they have a synchronized view.

Figure 3 shows the view from three different Microsoft HoloLens devices. In this shared augmented reality application, we display a fuel injection simulation data [8] where the scientists are interested in the design of the fuel injection subsystem. Since we do not have access to the actual fuel injection hardware at this point, we used a blue physical object in place of a fuel injector. The views are taken from the users A, B, and C standing around the holographic object as shown in Fig. 2. Just as the users are standing around a physical object, the view of the holographic object differ slightly based on the viewing angle from the Microsoft HoloLens. However, all the views are showing the same 3D holographic object being placed on top of the blue physical object in the real world.

Fig. 3.
figure 3

Views from three Microsoft HoloLens showing the placement of 3D holographic object on top of a blue cylinder in the actual environment. From left to right, User A, User B, and User C respectively as viewed from their position as shown in Fig. 2. (Color figure online)

Full size image

Although our setup has limited the roaming area of a Microsoft HoloLens to the tracking coverage of the OptiTrack, we believe the improved 3D holographic object registration ability contributes more to the usability of the shared augmented reality application. However, we can also expand the 3D tracking coverage if a larger area is needed for the shared augmented reality application by using a different type of sensor with more aerial coverage. Depending on the task requirement, if greater holographic object placement accuracy is needed, we can always use a 3D positioning tracking system with higher precision.

3.2 Comparative Study on Registration Error of Shared Augmented Reality Experiences: HoloToolkit Sharing Service Vs. OptiTrack Sensor Data Fusion

The main purpose of this comparative study is to determine if the shared augmented reality application based on our data fusion framework has less registration error compared to the shared application based on the HoloToolkit Sharing Service.

Comparative Study Design.

In our study, we used a completely randomized experiment design with replication and counterbalancing. Our study consisted of two factors and two treatments. We collected data from six male and 2 female participants from our research laboratory, and only one of them had prior experience with augmented reality. We divided the participants into four groups of two and applied a single treatment to each group. Each treatment contained two factors.

The two factors were the HoloToolkit Sharing Service shared experience, which we assigned as setup number 1, and the OptiTrack Sensor Data Fusion shared experience, which we assigned as setup number 2. For the treatments, the participants either experienced setup 1 followed by setup 2, or setup 2 followed by setup 1. Of the four groups, two of the groups experienced the HoloToolkit Sharing Service first, then the OptiTrack sensor sharing. The other two groups experienced the OptiTrack sensor sharing first, followed by the HoloToolkit Sharing Service.

For the experiment, the goal was to test how well an object registered in a shared user environment. Using the HoloToolkit Sharing Service, the first participant to connect saw the object first, then the second participant would join the shared augmented reality experience and sync to the first via the sharing service. Using the OptiTrack sensor sharing, the first participant would see the object, then the second participant to join would see the holographic object via the OptiTrack sensor’s communication of where the HoloLens device was in the physical world.

For both factors, the same holographic object of a fuel spray model was used. Once both HoloLens devices were connected to the shared experience, one participant was asked to place a physical object beneath the holographic object to mark where the object registered from his or her HoloLens’ point of view. The other participant was encouraged to give verbal feedback of how close the physical marker was to the holographic object from his or her own device’s point of view. The second participant was then given the physical marker and was asked to place it underneath the holographic object from his or her point of view. This gave both participants the ability to see where the other participant’s holographic object registered in the real world.

Comparative Study Results.

For the evaluation, the participants were encouraged to discuss among themselves on how near or far the holographic object appeared from the other participant’s. During the treatment, participants were asked to answer a short survey of questions following each factor, with each set of questions pertaining to that particular factor’s experience. The questions were the same for both factors. However, the results varied depending on which treatment the group of participants were assigned, determining which experience they encountered first.

The main question participants were asked was how close the holographic object showed up to the physical marker within the shared experience. The participants were asked to give a rating for how close the holographic object appeared to the physical object in the real world using a Likert scale from 1 to 5, with 1 being ‘not very close’ and 5 being ‘very close’. Each pair of participants was assigned a Group ID letter so that we could keep track of which group experienced which treatment. So for each group, we averaged the rating for each factor as seen by the chart in Fig. 4.

Fig. 4.
figure 4

Results from the comparative study showing average rating by group.

Full size image

As shown in Fig. 4, the average rating of each group rated the OptiTrack experience higher than the HoloToolkit experience. The most interesting result of the entire study, however, is the average ratings of all groups that took part in similar treatments. There were two separate treatments of the study. The first treatment was having the participants experience the HoloToolkit Sharing Service first, followed by the OptiTrack Sensor Data Fusion. The second experiment was having the users experience the OptiTrack Sensor Data Fusion first, followed by the HoloToolkit Sharing Service. We use setup 1 to represent the Sharing Service experience and setup 2 to represent the OptiTrack experience.

The main take away of the study is the rating given to the second setup the participant’s experienced, given the rating they gave to the first factor within their treatment. As shown in Fig. 5, the average rating of both groups rated the OptiTrack experience higher than the HoloToolkit experience in the holographic object placement. For participants that experienced the HoloToolkit Sharing Service first, the rating for that registration averaged a 3 considering the participants had no prior knowledge of sharing registration. When those participants then experienced the OptiTrack Sensor Data Fusion experience following the HoloToolkit experience, the average rating was 4.25 for the OptiTrack. This shows that the OptiTrack experience seemed to do better with registering the shared holographic object. This was a positive reinforcement that the OptiTrack Data Sensor Fusion performs better than the HoloToolkit Sharing Service. On the other hand, when the participants experienced the OptiTrack first, the average rating was only 3.5 for the registration of the holographic object. However, the average rating for the HoloToolkit experience then dropped to 1.75. This shows that having no prior experience, the OptiTrack experience still seemed to look appealing to users with how close the object registered, but also that it definitely did a better job than the HoloToolkit considering the significant drop off in scores for the second experience.

Fig. 5.
figure 5

Results from the comparative study showing average ratings by treatment groups

Full size image

When analyzing the results of this study, it is more important to observe the average ratings for each treatment as a whole rather than to directly observe the rating from 1 to 5 for the individual factors. As shown in Fig. 5, comparing the average ratings of one factor to the other given what treatment group the participant was assigned shows that using the OptiTrack Sensor Data Fusion shared experience does better to register holographic objects in a shared augmented reality environment. We do not claim to have solved the registration error in shared augmented reality experiences, but rather that we have created a better solution to registering holographic objects using multiple sensors.

4 Shared Augmented Reality Application to Support Mission Planning Scenario

Our shared augmented reality application supports mission planning scenario for multiple users to collaborate in a shared mixed reality environment. The main objective of the application is to enable the users to view and discuss key mission objectives and plans using augmented reality technology such that collaborators can not only view the same 3D holographic scene, but also interact with each other face to face while making eye contact. The mission planning application demonstrates an extraction scenario of a High Value Target (HVT) from an overrun building, with landing and extraction occurring using a sea route (ship). The team of soldiers arrives by ship and assembles at Assembly Area Alpha. The team then follows the predefined path behind the tree line and side of the building to Assault Position Bravo. The team then enters Objective Point through the side door of the compound and takes the shortest path to retrieve the HVT while avoiding patrolling hostiles. Once the HVT has been rescued, the team of soldiers exits the building through the back door, then makes their way to the extraction location. The team will board the ship at the extraction location and the scenario completes. Figure 6 shows the planned path denoted by blue dots showing landing and extraction path.

Fig. 6.
figure 6

Planned path in blue showing landing and extraction path. (Color figure online)

Full size image

The scenario is created in Unity using 3D assets and deployed to a Microsoft HoloLens device. While the building, terrain, and ship assets are all 3D models imported into the scene, the rest of the assets described in the scenario are created using 3D marker within Unity. Currently, the team of soldiers is represented by a blue diamond. Diamonds of different colors can be used to illustrate multiple teams engaged in the mission. The hostiles in the scene are represented by red diamonds that navigate along a fixed path both inside and outside of the compound to simulate aerial patrol. The number of hostiles may be changed and their paths can be manipulated depending on the mission planning scenario. The predetermined path from the landing point, through the building to the HVT, then to the extraction location is represented by a blue dotted line. As the team moves along the path, we illustrate the animation by drawing a solid line along the planned path. The HVT in the scenario is made obvious to the user by a large 3D arrow pointing down toward the location within the building. The arrow is initially grey when the scenario begins, then turns green once the team has successfully rescued it by passing by its location within the building. The extraction point is where the team meets the boat to leave the hostile environment and return to a secure location with the HVT.

Although the application currently runs with this specified scenario, there are many parameters that can be customized in Unity before the application is deployed to the HoloLens device. Depending on the planning scenario, the following parameters can be modified: the number of hostiles, where the hostiles are in the scene, the path of the hostiles, the number of team members, the team’s path from landing to extraction, how quickly the team moves along the path, how quickly the ship moves to the extraction point, and the colors of the team, hostiles, path, and target.

4.1 Stand Alone Application

In the stand-alone version of the Mission Planning Scenario application we created, the user is able to run a predefined scenario in a HoloLens device. On launch, a terrain model with a building and a ship placed on it will be shown. To the left of the terrain there is an instruction menu to help the user manipulate the scenario. On the instruction menu is a list of voice commands that the user can say in order to initiate different capabilities within the scene. The user can view the scene from different perspectives by physically walking around in the real-world environment as the scene remains fixed.

However, the user may also manipulate the scene by rotating, dragging, or resizing the scene by using the voice commands “Rotate Scene,” “Drag Scene,” or “Resize Scene” respectively. To rotate the scene, the user taps and holds, then moves the hand to the left or right and releases the taps to complete the rotation. To drag the scene, the user must also grab and hold, but can then move the hand in any direction. The scene will be placed at the new location where the user releases the tapped press. To resize the scene, the user must tap and hold, then move his/her hand up or down: up to make the scene bigger and down to make the scene smaller. The user’s voice command will initiate the specified manipulation technique until the user performs the corresponding action and then releases.

When the scene loads, many aspects of the scene can be toggled on or off depending on what the user wants to focus on. The voice command “Show Team” will toggle on viewing of the team that will be performing the rescue. This is currently a blue diamond that appears next to the boat at Assembly Area Alpha. The “Show Path” voice command toggles on or off the predefined path that the team will take starting from the ship into the building to rescue the HVT, and then to the extraction point. The path is a dotted line that changes to a solid line as the team moves from dot to dot along the path.

The “Show Target” voice command toggles on or off the HVT to be rescued by the team. The target will be a grey arrow pointing toward a location inside the building until the team rescues it, turning the arrow green. The “Show Hostiles” voice command will toggle on or off the hostiles in the scene. The “Run Simulation” voice command plays the animation of the scenario. As the team travels along the path, the hostiles will be moving back and forth, and the boat will make its way to the extraction point. The “Pause Simulation” voice command will pause the scene at any point during the simulation. This allows users to pause the scenario, manipulate the scene by toggling on or off something to view or by rotating, dragging, or resizing, then continuing to run the simulation. The last voice command is “Reset Units” which resets the entire scenario back to the beginning.

4.2 Shared Augmented Reality Application

In the shared version of our application, we have a server application running on the same machine that hosts the OptiTrack’s Motive software, and a client application that runs on the HoloLens devices. In order to run the application, the server must first be running and connected to the Motive software data streaming to receive the 3D positional tracking data. Then, the client application running on each independent HoloLens may be started and the client application will connect to the server over the network.

When the client connects, the user will see the predefined scenario in a shared environment. All users will see the scene in the same position and orientation at a tabletop height in the real-world environment, in the center of the OptiTrack cameras’ capture area. Underneath the scene is a holographic cylinder that rests on the ground as if the scene were sitting on a real-world object like a table. This allows users to physically point out objects in the scene and discuss with each other naturally as if they were standing around a table in the real world.

The predefined scenario is set with the team in place, the path mapped out, and the target set. There are no voice commands in the shared application as the users will generally be in a close enough vicinity that a voice command would register in multiple users’ HoloLens devices. Instead, any user has the ability to begin the scenario by simply performing an air tap gesture while gazing at the cylinder. The air tap registered by one of the clients sends a message back to the server, which then broadcasts the run simulation command out to all connected clients. The team and boat will begin moving and the scenario will play out in unison for all clients. Users are free to move around in order to inspect the scene from different angles and viewpoints, and to discuss with the other users what they are all seeing.

5 Conclusion and Future Work

We investigated using data from different sensors to improve 3D holographic object registration for a shared augmented reality application in our sensor data fusion framework. We used the technique to design a working shared augmented reality application to support collaborative mission scenario planning.

In a sensor rich environment, various sensor data can be used to derive the necessary information about the 3D environment needed to improve a shared augmented reality experience. Similar to the data fusion work with information from multiple data source to generate situational awareness, our existing framework allows us to use data fusion techniques on data from different sensors to enrich our augmented reality experience. We plan to expand our work to include sensor data from networking devices, motion detection sensors, and multi-spectral cameras. Another research area is to build the algorithm to determine when to update or resynchronize data from which external sensors.