Evaluating Virtual Reality Simulations for Construction Safety Training

1 Introduction

One of the most challenging tasks for the construction and industry sector to date is to increase occupational health and safety and reduce accidents on construction sites. For 2020, 97 deadly accidents were reported for the German construction industry [2]. Factors that lead to occupational accidents are diverse, including: inattentiveness or distraction [14], [32], fatigue [11], mismatch of competence [10] or inexperience [11], to name a few. In addition, construction workers are not prepared for dangerous situations in the typically very complex and highly dynamic work environment. Consequentially, it is hardly possible to experience hazards beforehand in a safe environment and train adequate and safe reactions. In order to avoid accidents and hazardous situations, construction companies use various approaches to solve this problem, for instance, by applying safety inspections, safety rules or even penalties for non-compliance with safety rules [27].

Figure 1

Starting scene: viewpoint of the trainee’s participant of the VR simulation in the foreman’s office.

Another method that is widely used in the industry and focusses on workers’ safety trainings for accident prevention. These training methods include hands-on practice, not just theoretical safety workshops and are a promising approach for workers as knowledge has to be applied directly. We suppose that the more direct an experience – e. g. a near-accident – is related to the lesson to be learnt, the more it results in a learning effect that directs an individual’s behavior [15]. However, in the case of training that helps to avoid safety risks, learning must not be based on real accidents. With the use of Virtual Reality (VR), such training can take place in immersive virtual environments that also allow for experiencing hazards and safety risks. Therefore, this paper addresses the research question to what extent VR can contribute to virtually experiencing accidents or near-accidents that have a similarly instructive effect as real events.

Several studies have already shown benefits of VR trainings, like, for example, an increased attention and concentration increasing knowledge acquisition of study participants compared to conventional methods [26], [37]. In comparison to theoretical lessons, it was observed that participants were able to maintain focus during a VR safety training for a longer period of time and thus create a stronger learning effect [37].

However, to our knowledge, there is only minor research available that examines the question whether VR can be used to provide an experience of accidents that has a sufficient training effect of increasing awareness for safety risks. For example, in [22] a collaborative construction safety VR system for experiential learning is proposed that allows to understand accident causes and their prevention. To evaluate the system, the researchers focused on usability of the system and task performance but did not assess the knowledge acquisition of each student via the VR experience. In our study, we seek to answer our research question with an exemplary study design of an available VR-solution.

In the present research, a VR based safety training simulation to work with angle grinders was evaluated with 14 machine operator trainees and two instructors (Figure 1). We analyzed the perceived usability of the system, the participants’ user experience and learning of safety knowledge. For a deeper understanding of participants’ decision behavior during the simulation, semi-structured interviews were conducted afterwards with each participant.

2 Virtual Reality Trainings for Construction Workplace Safety

In the construction industry, considerable research interest exists in developing Virtual Reality (VR) simulations and applications for safety education, safety and health training and hazard identification (e. g. [25], [34]). One of the major advantages of VR is the potentially high immersion to experience hazards and the handling of dangerous work equipment without being exposed to any real danger [13]. Thus, with virtual environments, on-the-job-training or experiential learning can be done in risk-free environments while still feeling a high amount of presence in the scene.

With the ability to experience hazards in a safe environment, there are many studies on how VR simulations can reduce accidents and increase safety in the construction industry (e. g. [13], [16], [25], [43], [44]) or other areas like pedestrian safety or evacuation tasks (e. g. [19], [20], [30]). For example, it was examined how VR can contribute to safety promotion in construction tasks, by designing effective training modules that allow trainees to become familiar with all kinds of hazards in the work environment [46]. VR is also used to investigate very specific use cases of construction site work, for example, hazard detection on construction sites [34], working in heights [35] or examining workers‘ inattentiveness during construction tasks [17], [18]. A prototype of a VR interactive training environment that enables players to experience and reflect on consequences in a VR construction site is described in [12]. Experiencing hazards in VR simulations can also bring out memories of previously experienced hazards and trigger reflection processes [8].

The use of Building Information Modeling (BIM) in the construction industry facilitates to use daily accurate 3D-representations of construction sites for VR scenarios [7], [45]. Thus, future use of BIM data could enable to analyze accident scenarios virtually, or to walk through potentially hazardous environments beforehand [40]. For an efficient development of VR environments for engineering and construction, solutions incorporating BIM into the unreal game engine have been proposed in [13].

Evaluating a VR simulation for medical emergency training, Lerner et al. [23] have compared pre- and post-knowledge of participants after experiencing the VR simulation. They also assessed the Igroup Presence Questionnaire [39], the System Usability Scale [4], and the Training Evaluation Inventory (TEI) [36], in addition to other measurements. The results of their study have shown a significant correlation between the experience of presence (in the scene) and the perceived training effectiveness. The system usability of the VR training was rated as sufficiently with a score of 65.56 points. However, the results did not show a significant difference between the pretest and posttest of the knowledge test.

Although a considerable body of research has investigated the use of VR in various construction settings for work safety, less attention has been paid to acquiring safety knowledge about construction tools through VR simulations. For example, Le, Pedro & Park [22] focused on providing a collaborative VR system for students and teachers using a construction environment to understand and explain general safety risks in construction work. In their evaluation they assessed the subjective views on the usability of the system, the participants’ task load and compared the system between collaborative use and a single user, but did not compare the individual knowledge acquisition of each user. The results showed that the ease of use was rated as good, the collaboration function was rated as very useful, but that educators criticized the creation of scenarios as time consuming. Some research contributions in VR safety training highlight the development process of VR simulations or focus on technical implementations without evaluating the prototypes for teaching safety knowledge. For example, Filigenzi & Orr [9] have shown a VR system for mine safety training, but did not conduct an evaluation of the system. Similarly, Pérez et al. [33] describe the development of a VR safety training for offshore oil platforms but have not conducted an evaluation of their prototype. The same applies for [21], where a VR construction safety training is proposed and the potentials of such system are highlighted, and for [31], where a VR experience-based safety training for chemical plants is proposed.

Sacks et al. [37] have proposed a CAVE (Cave Automatic Virtual Environment) construction safety training and compared it to conventional methods in a ‘between-subjects’ experimental design. To evaluate the system, participants did a safety knowledge test as baseline measurement and were then divided into two groups: one group received the VR training, the other group received conventional classroom instructions. In the VR training, the group participants simultaneously experienced several accident scenarios of an avatar in a ‘third person’ view. After the training, both groups repeated the safety knowledge test and filled out an experience questionnaire. The results have shown significant advantages of the VR training for stone cladding work and cast-in-situ concrete work, but not for general construction site safety. Additionally, the learning effectiveness was higher in the VR training and participants in the VR group did maintain a higher level of alertness and concentration compared to the group with the conventional training.

In our research, we present the evaluation of an interactive VR safety training in which participants individually experienced a construction site scenario and received safety aspects about working safely with angle grinders, without providing additional feedback or learning materials any other than given by the simulations’ content. To finish the simulation successfully, participants had to consider multiple safety aspects while they were pursuing the task to cut an air shaft. Whenever a participant made a mistake or forgot a safety measure, an accident happened and the simulation restarted, giving participants visual feedback about their mistake. In addition to the visual feedback, the participants received an explanation of the foreman as non-player character (NPC) (Figure 1).

The aim of the study was to see how participants perceive such VR simulation and whether knowledge about occupational safety while working with angle grinders is acquired incidentally by experiencing the simulation.

3 Methodological Approach

Figure 2

Inside the VR simulation, participants had to choose between a safe angle grinder (on the left) and an unsafe one (right).

In this study, we used a VR simulation in which participants had to pay attention to their safety and the safety of co-workers in a construction building. Inside the VR simulation, participants were given a task by a foreman (an NPC) to make a cut with an angle grinder into an air shaft (see Figure 5). In order to accomplish the given task, participants first had to choose the correct personal protection equipment (PPE) and a safe angle grinder, then had to walk across the construction site via a specific path to reach the air shaft. During the simulation, participants could experience several hazards or accidents and had to make the right decision for their own safety:

1. Choosing the correct personal protection equipment (PPE), by deciding between a helmet with or without a full-face shield. Since not only sparks can fly when working with angle grinders, but also cut-off parts, safety goggles or face shields should be used in any case.

2. Selecting a functioning and safe working tool, by deciding between an angle grinder with or without guard and handle. Depending on the material to be cut, angle grinders can become jammed in the material. This can result in strong recoil, so angle grinders should always be used with both hands. In addition, there should always be a wheel guard on an angle grinder, which should be positioned with the back to the body, as this ensures that in the event of a shattering cut-off wheel, the parts do not hit the body.

3. Choose a safe path between the starting point of the simulation (supervisors’ office) and the area in which the task had to be fulfilled. Due to parked material, traffic and other workers, construction sites can be confusing. To avoid putting yourself in danger, a safe path to the worksite should be used.

4. Keeping the personal workspace clear of other workers and avoid to walk into or through the workspace of other co-workers (see Figure 4). It should always be ensured that one’s worksite is free of other workers who could otherwise be struck by flying or falling parts. Likewise, one should not enter the work area of other workers unannounced, as there is also the risk of being hit by flying parts.

5. Pay attention to the traffic on the construction site and do not get run over by a forklift.

Figure 3

Schematic overview of the virtual construction site, the safe path participants had to take and the placement of hazards.

Figure 4

Example of an unsafe path, as the participant (wearing the safety helmet) would have to walk through the workspace of a co-worker that is using an angle grinder.

To play the simulation, participants were equipped with an HTC Vive Pro Headset and two controllers. In the game, their player character was only equipped with safety gloves. After a brief introduction by the supervisor/foreman in the game, participants could begin the simulation. The playtime varied between 3–15 minutes, depending on each users’ decisions and necessary trials to complete the task. If a participant made a mistake, e. g. was run over by a forklift truck, the game started from the beginning and the supervisor game character explained what the participant did wrong.

3.3 Study Procedure

Prior to the actual testing of the simulation, several pre-runs of the study were iteratively conducted with professionals from the construction industry and research. Based on the observations and comments during these tests, adjustments were made to the simulation and to the study design. These five pre-runs were mainly intended for quality assurance purposes and to resolve inconsistencies and inaccuracies in the content of the simulation and its incremental development. For example, the hazard of an oil puddle was removed as it was overlooked in every pre-run and the resulting fall over (within the simulation) led to motion sickness symptoms (see Section 3). Adjustments were also made to other hazard areas to ensure that they were presented correctly in terms of content and, most importantly, that they were clearly understandable to the participants.

The main study was conducted on two consecutive days. On the first day, the participants were informed about the general course of the study without going into the content of the simulation. They received the privacy statement and were informed that they could withdraw their participation in the study at any time. Additionally, they also got the opportunity to ask questions in case they had any. Each participant then completed the questionnaire with 38 items about the general occupational safety on construction sites. At the end of this first session, the participants received an individual time slot for their participation on the next day.

Figure 5

The simulation was successfully completed when all safety measures were followed and participants started cutting a ventilation shaft.

On the second day, each participant was given a brief explanation of the technology and the controllers. There were no further instructions than that they will have to fulfill a task during the simulation. Before they started the simulation, each participant could try out the controllers in a short VR tutorial. During the tutorial they learned how to move around in the VR environment by using the controllers and how to pick up and use a power tool. To this point, participants had the possibility to ask questions if anything was unclear. When they were ready the simulation started in the office of a supervisor/foreman on a construction site. The avatar of the supervisor told participants their task in the simulation. Participants had to choose between two different helmets which differed only in having a visual protection or not, and between two different angle grinders, where one of the angle grinders was missing the side arm. Next to the exit door of the office was an overview map on which the participants could see their place of work for the task.

The simulation could only be completed successfully when all necessary safety aspects had been considered. The participants were then relieved of the two controllers and the head-mounted device and were asked briefly about their general state of health. They then went with the investigator to a room next door and completed the questionnaire on general occupational safety again. Following this, participants filled out the SUS and the UEQ. After completing the questionnaires, participants were surveyed in a semi-structured qualitative interview.

4 Results

The data analysis focused on three different aspects. First, the general usability and user experience of the simulation was determined to understand whether the usability of the simulation is sufficient and how the participants evaluate the VR simulation in terms of user experience. Second, the questionnaire on general safety was used to check to which extent short-term learning effects could be identified. By analyzing the results of this questionnaire, we can examine whether the VR training has led to any knowledge acquisition at all. Finally, the qualitative interviews were evaluated in order to obtain additional information about the acceptance, user experience, usability and the content of the simulation and the system.

The data collected by the four questionnaires (pre and post safety questionnaire, SUS, UEQ) and the semi-structured interviews of all 12 participants and the 2 experts, were analyzed in regard to learning effects, usability and perception and preferences for a specific learning method for safety training.

Methodically, the two general safety questionnaires were evaluated comparing the pre and post results, calculating mean and using a paired t-test to estimate the statistical significance, while Cohens’ d was used to predict the correlation and the effect strength. Here the analyses were realized separately for the overall 14 items and one multiple-choice question concerning the content of the scenario and the whole questionnaires, counting 38 items.

4.1 Usability and User Experience

For the analysis of the systems’ usability and the user experience, the SUS and UEQ were used and combined with participants’ statements made during the interviews. The results of the SUS scores were calculated as proposed in [4] and show a score of 78.7 (SD=11.6), which can be described as highly acceptable on the acceptability range and “good” on the adjective rating [3]. A SUS score over 68 is considered above average and a score of 78.7, considering percentile rankings indicate a very positive perception and usability. During the interviews, participants supported the usability rating with further comments about the simulation. For example, participants noted that controlling the player in the simulation was easy after trying it during the tutorial. No other problems were mentioned than to memorize which button resulted in which action, but even that was considered as acceptable and would vanish with frequent use.

The results for the UEQ show good to excellent rating of the simulation for each category. The overall attractivity reaches a score of 2.06 (out of −3/+3), the pragmatic quality (perspicuity, efficiency and dependability) is scored at 1.67 and the hedonic quality (stimulation, novelty) reaches a score of 1.95 (an overview of the results for each individual scale is shown in Figure 6).

Figure 6

Results of the User Experience Questionnaire (UEQ), showing that participants gave the VR simulation positive ratings for attractiveness (2.06), perspicuity (1.88), efficiency (1.6), dependability (1.52), stimulation (2.21), and novelty (1.7).

The qualitative data supports the findings of the UEQ. The interviews showed that most trainees would prefer VR training over theoretical training and gave a very positive assessment. During the semi-structured interviews, all 14 participants rated the visualization of simulation as very positive, while 11 of 14 participants experienced the simulation as very positive and realistic overall. Considering the training of work safety aspects, 9 out of the 15 participants acknowledged the current version simulation as helpful. Especially picking the right PPE and the possibility to virtually experience work-hazards was seen as highly effective. Additionally, 10 participants mentioned they felt safer after exploring the simulation, especially concerning general safety aspects.

Finally, all participants managed to solve the task on their own, without any interfering comments from the experiment supervisor, making the simulation itself usable without further external supervision.

5 Conclusion and Discussion

The results have shown, that the VR scenario for training received great acceptance and was seen as very motivational by the participating trainees. However, great efforts were required to avoid inaccuracies and inconsistencies in the technical implementation of the simulation, as the five pre-runs had shown. The main study showed that the simulation was very positively received by the participants. It was also considered to be an enjoyable experience and we found slightly better results in the safety questionnaire after playing the simulation. Raising working safety awareness is important, especially in high risk areas like construction work. Thus, adding VR simulations to safety training not only supports existing training but also increases consciousness for work related accidents.

In this regard, the hypotheses that construction workers benefit overall in construction safety training through VR simulations can be confirmed through the interviews and quantitative data. Findings show significant learning improvements for working with angle grinders when comparing pre- and post-results of the safety knowledge questionnaire but no significant increase in the results for general work safety on construction sites. Due to the chosen study design, we cannot exclude the possibility that priming effects on safety knowledge for working with angle grinders may have occurred through the pretest questionnaire. However, if such a priming effect took place, it does hardly explain the difference regarding the effects on work with angle grinders vs. general work safety.

The evaluation of the VR simulation could be extended to make a comparison to conventional learning materials. For example, it is conceivable to compare experiential learning via VR simulation with other learning materials, for example, with respect to experience, motivation, cognitive load, and knowledge acquisition. However, it was demonstrated that the attractivity and usability in users’ perception increases the awareness of participants, hence the efficiency, backing former findings of [37]. Conclusively, many participants chose VR as a preferred method over theoretical trainings, backing the findings about dissatisfaction with existing models of training [24], [42]. The overall positive assessment and desire for further development of the scenario shows the importance for further investigation and implementation of VR as a training method.

Since our final sample included only 12 participants, the results are hardly statistically representative. However, as the training should primary be used in professional training, our sample consists of participants of the considered target group. Additionally, the sample size is comparable to other studies in the same area of research [5]. In addition, the sample size is quite appropriate for usability tests. Only the learning effect should be treated with caution and further research in the field of VR based experience learning should be carried out.

In the VR simulation, the choice of the selectable PPE was very limited (only a helmet with or without visual cover) and the VR character was already wearing safety gloves, resulting in very short play durations for some participants. To extend the simulation, it would be useful to offer much more PPE, including some that is not necessary for the particular scenario. To minimize the memorization of the scenario on repetitions (after each accident), the dynamics could be adjusted automatically on each repetition, for example, by varying the number of other workers in the environment, varying the arrangement of the PPE, or placing hazards in different spots. The adaptations would allow the simulation to include multiple decision-making tasks, encouraging participants to think and reflect on their decision even more. Additionally, multiplayer scenarios would be conceivable. Either by several participants working simultaneously on their own tasks in the simulation and having to pay attention to their own safety and that of the other workers. Or also for collaborative working on construction tasks.

Furthermore, the training contents need to match the real-life experiences completely, as inconsistencies during the play was seen as confusing for participants. As a result of these inconsistencies, there is a risk that participants will acquire incorrect knowledge if the simulation content does not fully follow the country’s current safety regulations. Therefore, to prevent the simulation from teaching inaccurate knowledge, we recommend that safety regulations and recommendations should be included in the design of the scenario from the beginning. Additionally, during the development process, the implementation of these regulations into the simulation should be evaluated iteratively.

Finally, we want to address the practicality of such VR simulations for experiential learning purposes. While an at least short-term learning effect could be shown, the development of this VR simulation needed approximately 6 months development time in multiple iterations with a small team of three developers. In addition to the technical implementation, the development also included the selection of a use case (angle grinder), the development of typical accident hazards within the use case and the creation of a scenario for the simulation. Furthermore, it should be considered that the required skills for VR developers are quite high. In addition to programming expertise, they also have to be proficient in the use of game engines to ensure that the VR simulation scenarios are well implemented both audio-visually. Despite the positive perception of the VR simulation by the participants, the development effort clearly minimizes the practicability of such simulations. However, apart from the VR hardware, VR simulations are easily scalable to be distributed on a larger scope, thus having the potential to be used as experiential trainings in the industry.