Keywords

1 Introduction

Simulators, providing a virtual environment to users are nowadays available in different varieties and range from low-cost gaming simulators to highly realistic vehicle or industry simulators, such as professional car or flight simulators. Many development teams or researcher do not have access to high-end simulators. This might not be needed or cannot be afforded in many cases. Nonetheless, we argue that only using simple PC-based simulators might limit options for research, design and development teams when testing new concepts and ideas for industrial vehicles.

Generally, simulators provide advantages in terms of reproducibility, standardization and controllability of scenarios and tests. A simulator replicates scenarios that might be difficult, expensive or even risky to reproduce in reality, such as a dangerous driving scenarios, where a driver would be physically at risk. Furthermore, it allows controlled simulation environments that are not affected by wind, weather and other external circumstances [7]. Additionally, the use of simulators and interactions with them can be evaluated without placing humans or physical objects in a real environment or before the real environment is ready, for example when evaluating improved operator environments.

These facts would argue for using a simulator in test and design scenarios. However, despite the progress in hardware and software used for simulators, there are still limitations. Industrial simulators are usually built with computer monitors or projection solutions that can be space consuming, require long setup times and pose high costs. Therefore, initial experiments, user evaluations or even trainings are often performed on simpler simulators. Fully featured simulation environments are used in later stages of a project, when previous stages indicated positive results. This limitation and way of working results in increased costs as well as it limits early evaluation of interaction with the technology or interface.

Another alternative when making the simulation is to use virtual reality (VR) glasses, which provide the ability to create an immersive virtual reality. Simulating a virtual environment in which the user can naturally look around by moving the head to interact with the virtual environment can be important in many scenarios, such as the simulation of complex industrial vehicles (cranes, excavators, fork lifts, etc.) or even normal driving scenarios (e.g. overtaking, parking, etc.) [13, 16]. A mayor limitation with virtual reality is when physical artefacts like sliders, knobs and interior design of a real world prototype have to be used while performing in the virtual environment, e.g. mixed reality.

This paper introduces a low-cost mixed reality simulator that challenges many of the above-mentioned constraints. Our solution is easy to install, low-cost, consists of off-the-self hardware and a virtual environment that was created with the Unity game engine. In our prototype, we create a mixed reality environment while using physical controls and a projected virtual scenery. Our technology can be used in small setups where virtual windows or screens are in front of a user or to create larger CAVE-like simulations.

Furthermore, we evaluated if our mixed reality simulator provides the basic ability to be used in a scenario such as for industrial vehicles. Therefore, we created an industrial vehicle simulator resembling the physical controls and virtual representation of an excavator on a virtual obstacle course. In such environments it is of higher importance, compared to ordinary on-road cars, to be able to look up and down and to see the environment around the vehicle. In our simulator, users can explore the virtual environment freely by naturally moving their head in a CAVE-like setup.

The contribution of this paper is a low cost mixed reality simulator that can be used for testing and designing human machine interface (HMI) applications, such as evaluating new concepts to interact with vehicles, for education or for user training with industrial vehicles.

2 Related Work

Simulators have been used in many vehicle research areas to evaluate user behavior and HMIs as well as for education. These can range from specific replica simulators costing hundreds of thousands euros [24] to low cost simulators [3] and desk simulators [23]. The development of 3D engines with physics simulation and collision detection, as well as easy to use editors, also make it possible to build custom simulators [6]. For many purposes a more light weight simulator is sufficient, “as our ability to fill in the gaps to create strong cognitive representations has clear potential as an alternative to modeling every last detail of the space” [28]. In an on-road driving simulation, for example, it may be enough to look at a PC monitor to display the simulated environment in front of a car and to use a gaming steering wheel as input device. However this approach offers a very limited field-of-view (FOV), limiting simulations where operators need to move their head or body. This is especially a problem when interactions need to be tested that require to see the surrounding environment.

The interest to build virtual and mixed reality display systems [5, 25, 26] has increased, as new products are introduced to the market. Most of the see-through products offer a near-eye solution with a limited field of view, a constant focus, single eye usage (no stereoscopy), and limited depth of field. Many near-eye solutions come with a great deal of optical complexity in design, e.g., Google Glass [22], or Microsoft‘s HoloLens [19]. In [10] an additional specially made contact lens needs to be used to see the content. Thus, the users are having the challenge of interacting naturally with the physical world due to these optical limitations.

By disconnecting the user from the real world, the mentioned limitations are avoided. This is achieved via opaque wearable stereoscopic head-worn displays, e.g., Oculus Rift [8], also known as virtual reality glasses. Nonetheless, the challenge for real-life use cases remains the same or is even higher, as real world content now must be replicated into the virtual world [18].

While simulators are a great tool for assessing different scenarios and measuring driving performances, participants often face the problem of suffering from motion sickness [7]. This observation has been made in various research areas, including vehicle simulators [13] and flight simulators [17]. Motion sickness appears in mixed-reality and VR simulations respectively. Research showed, that the time spent in a simulator affects the likelihood of simulator sickness and that older participants have a greater likelihood of simulator sickness than younger participants [4]. However, participants react differently to simulation environments. Therefore, we found it important to get an indication on how our simulator behaves in this respect.

The use of laser scanning pico-projectors offers an interesting alternative, as these do not require any optical components to focus on the projection surface and the amount of pixels displayed stays constant with the increasing distance between the projector and the screen. Additionally, it comes with a coin size light engine [9] with further possibilities for miniaturization, thus making it even more wearable. The image qualities as well as the use of reflective material has been investigated before [2, 12]. Image projections can also be made onto non-reflective surfaces [20, 21]. However, in our setup we use a reflective material as this enables to mix physical controls with the virtual scenery without distortion and in different lighting conditions.

Wearable laser projectors, as we use it in our research, were used by Harrison et al. before [11]. They used a shoulder mounted projector combined with a depth camera to project images to non-reflective surfaces and to allow gestural interactions. In our approach we have further developed the idea of using a laser projector as a head-mounted mixed-reality device using a retro-reflective surface as screen. This technology was used before for motion capture acting support applications and for gaming tests [1, 15]. Unlike other systems like e.g. from CastAR [14] or other research [27] this system does not require multiple projectors. Thus, problems originated from using multiple projectors such as the keystone or image registration effect are not an issue in our system.

3 Simulator Setup

The goal with our research was to build a low-cost simulator that allows to test new control concepts and design ideas in a fast and prototypical way. Therefore, it was of importance for us that the real and virtual world can be controlled and seen at the same time. We built a mixed reality simulator that allows to steer an excavator over an obstacle course by using a conventional joystick and wireless keyboard. Our mixed reality simulator setup consists of a head-mounted projection display, a room coated with a reflective cloth, input controllers mounted on a chair and software to drive and display the simulation.

3.1 Head-Worn Projection Display

Our head-worn projection display system, as shown in Fig. 1 consists of several off-the-shelf hardware: (1) a stripped-down laser pico projector, SHOWWX+ from Microvision, Inc., with external battery pack (2) a Samsung S4+ smartphone, (3) retro-reflective cloth covering the walls of our simulator room (can be seen Fig. 2), and (4) a headband with 3D printed housing holding the equipment.

A pico projector acts as a light engine in our head-mounted projection display. This has the ability that users who are wearing the prototype do not suffer from any key distortion effect, even when the reflective cloth used as a screen is distorted or not perfectly flat. This enables for a faster screen or CAVE-like setup by still allowing a clear and distortion free image. This lies in the optical abilities of the laser projector. The maximum native resolution of the projector is \(848px\times 480px~@60\,Hz~vsync\) in size by a light emission of 15 lumen. To increase the image size, we attached a \(180^{\circ }\) fisheye lens which allows for a field of view of roughly \(83.6^{\circ }\) x \(47.5^{\circ }\).

For our mixed reality application, we used the gyroscope sensor of the smartphone to look around in the digital environment and connected the pico projector through a MHL adapter to the smartphone. The overall system, has an expected uptime of 3 – 4 h.

Fig. 1.
figure 1

Picture showing the head-mounted projection display including a smartphone, laser projector with battery pack and 3D printed housing

Full size image

3.2 Projection Room

The simulator room was coated with a high-gain retro-reflective cloth no. 6101 from RB Reflektör. The cloth is used to reflect the projected light back to the source and has a high light gain. This effect allows users standing close to the projector to basically see a very bright image. We placed the reflective cloth in a \(4\,\times \,3\) m room with a height of 3 m and also covered the floor in front and around the user. In this room we placed a rotatable chair and mounted a joystick and bluetooth keyboard on the chair. The setup of the room and the chair can be seen in Fig. 2.

Fig. 2.
figure 2

The left-side picture depicts the simulator room while the right-side picture shows a user performing in it. The pictures also show components used for the simulator: 1. Bluetooth keyboard, 2. Joystick, 3. Reflective cloth, 4. Head-mounted projector, 5. Simulation projected to the screen

Full size image

3.3 Architecture and Software Description

Figure 3 provides a general overview of the technical components and the implemented architecture of our industrial simulator setup. The smartphone holds the digital simulator environment, developed in the Unity game engine version 5.2.3. The built-in gyroscope sensor of the smartphone is used to be able to look around in the digital environment with natural head movements. Moreover, we use the smartphone as a connection hub for the input devices and as a computation unit to show the digital environment through the connected projector. As joystick we used a Thrustmaster T.16000M which is connected to an Intel Compute Stick that runs a self-developed program converting joystick events into an XML format which is then sent via WIFI to the smartphone. Keyboard inputs were sent via Bluetooth to the smartphone. This data was thereafter used in the Unity engine to control the virtual environment.

Fig. 3.
figure 3

Architecture overview of the mixed reality simulator

Full size image

4 Functionality Test

For our informal functionality tests and to get a first impression from users about our industrial simulator, we conducted a test with 21 users (4 female and 17 male). Three testers were in the range of 20–25 years, 9 users were in the range of 26–35 and 9 were over 35 years old. Only one user had driven an excavator before; 8 testers have not experienced a wearable projector or display before and 13 tried it before. Moreover, 8 out of 21 users tried a vehicle or industrial simulator before.

To evaluate, if our mixed reality prototype could be useful as a simulator, we also built a PC-based simulator that held the same digital environment and controllers (joystick and keyboard). Figure 4 shows the PC-based test setup.

Fig. 4.
figure 4

PC-based simulator, with Bluetooth keyboard and directly connected joystick

Full size image

The user tests were conducted on one day and each user was given an introduction to both prototypes, its functionality and a minute of time to try out the controls. Users started with the PC-based simulator and performed an obstacle course where instructions were given during the test. Thereafter, the users performed the same tasks again in the mixed-reality simulator. We chose this order so that users were able to get familiar with the controls and the tasks at hand in a more familiar PC-based simulator environment. When testing the mixed reality simulator, users already had an understanding on how to control the excavator and what the tasks were.

Driving the excavator was set to the W, A, S, D keys of the keyboard and steering the rest of the excavator was performed via the joysticks axes and buttons. Moving the lower excavator arm up and down was set to the vertical axis of the joystick, moving the turret of the excavator to the horizontal axis, moving the upper arm was controlled by twisting or turning the joystick around the z axis and finally two buttons on the top of the joystick controlled the bucket movement of the excavator. These controls do not fully resemble the controls of a real excavator. This was intentionally chosen, as new control and design ideas can be tested with our mixed reality prototype. The combination of choosing a keyboard and a joystick might be rather unconventional in a real vehicle but fits to our concept of exploring new control and design concepts for heavy machinery. This might especially be interesting as users and gamers are used to such controls. Nonetheless, we did not focus on exploring this aspect in this research further.

An overview of the obstacle course is given in Fig. 5. The general tasks were to drive along the track, over and through different obstacles without hitting objects or placed traffic cones. We placed 2 cube towers on the obstacle course where users had to look up and use the excavator to push down the top 2 cubes of each cube tower. This task was especially meant to test the controls of the excavator and to evaluate the difference between the two simulators. Completing one obstacle course in a simulator took about 5 min, the whole user test for one user took 15 min to complete.

Fig. 5.
figure 5

Top-down view on the digital obstacle course with colored key elements

Full size image

The user tests were also videotaped as reference material and as another source of data collection to evaluate the user reactions and comfort or discomfort while using the prototype. In addition, a questionnaire was filled out by the users to help evaluating their experiences with both prototypes.

In the questionnaire we asked the users three for us essential questions: “Which of the simulators felt more realistic?”, “Which of the simulators felt more immersive?” and “Which of the simulators felt more natural to use and explore?”.

Figure 6 shows the results of these three questions. From the answers, we can see that for all three questions, the users are indicating that the mixed reality simulator offers a more realistic feel.

Fig. 6.
figure 6

Evaluation results for question based on immersion, realism and naturalness of use

Full size image

Moreover, we asked the users if they experienced a feeling of nausea or discomfort while using the mixed reality simulator. The answers were fairly distributed and showed that 42.86 % of the users had no such feeling, 23.81 % felt a light feeling of discomfort, 4.76 % had a mild feeling of discomfort and 28.57 % perceived a clear feeling of discomfort. These results are depicted in Fig. 7. Previous research has performed user tests with a car simulator using Oculus Rift DK2 VR glasses and tested 25 users [13]. They found that 84 % of their users felt motion sick in their tests. Our test results seem to show fewer issues with motion sickness. Nonetheless it needs to be tested with VR and even AR glasses in future research to solidify our findings.

Fig. 7.
figure 7

Evaluation results for nausea and discomfort of the mixed reality simulator

Full size image

Furthermore, we asked the user to rate their effort to complete the tasks in the two simulators. This indicated that it was slightly more difficult to complete the tasks in the mixed reality simulator, as can be seen in Fig. 8.

Fig. 8.
figure 8

Evaluation results for question based on the perceived efforts to complete the user tasks

Full size image

Testing our mixed reality simulator has shown that the general perception was positive and that users showed an indication of preference towards the mixed reality simulator in comparison to a PC-based setup. During the user tests, we observed that the users were quite focused on the tasks at hand and the, for them, rather unknown way of controlling an excavator. Our mixed reality setup supported the users, as the real world controls were visible and could be touched and tested naturally.

5 Conclusion

In this paper, we have shown a low-cost mixed reality simulator for industrial vehicles. The simulator was built from off-the-self equipment and allows for flexible scenarios, e.g. CAVE-like or in-vehicle setups. Our simulator uses a head-mounted projection display with a single 15 lm pico projector and does not require a cable connection to a stationary processing unit. Perceived images are focus-free, bright and distortion-free and the perceived field-of-view is approximately \(96^{\circ }\) in diagonal. The vision of the users is not occluded and no hardware is placed in the users’ field of vision. To evaluate our mixed reality simulator, we performed user tests with 21 participants and introduced a PC-based simulator using the same digital environment and controls as a baseline of our evaluations. The user test results indicated that participants had a more realistic, natural to use and immersive experience in the mixed reality simulator in contrast to the PC-based simulator. We observed that the ability to naturally look around in a CAVE-like large screen environment by still being able to see the real-world controls was beneficial to achieve these results.

In our view, we see the short set up time and low-cost of our mixed reality simulator prototype as a way to allow testing or developing prototypes and implementing design ideas for industrial vehicles or even other areas. Our tests were performed in a CAVE-like room but could even be installed inside a real vehicle cabin and combined with the controls of the vehicle. Therefore, we see our prototype as a basis for further research and a way to introduce simulators to different stages in designing and testing vehicular concepts.