Evaluation of an Augmented Reality Instruction for a Complex Assembly Task

1 Introduction

In addition to considerable opportunities for the economy and small and medium-sized enterprises due to increasing digitization [8], [18], there are also fundamental changes in the work environment [1]. Advancing globalization, changes in market requirements and individual customer needs are leading to an increasing number of variants and an increasingly complex production organization [28]. Growing product individuality leads to decreasing batch sizes and shorter product life cycles [23]. The resulting frequent changes in tasks lead, for example, to an increased frequency with which employees have to deal with new content [14]. Due to the new challenges, the flexibility required to adapt to new variants becomes increasingly important [4]. Due to this, manual or partially automated assembly is often used in order to take advantage of the high flexibility of humans [23]. The high number of variants results in an increase of components and tools in the assembly, which means a high demand of the employees [10]. Employees often only have access to assembly instructions in the form of assembly drawings or work instructions in text form, whose comprehensibility is insufficient in many cases [23]. The paper instructions used are often unintuitive [26] and frequently contain redundant and confusing information [17]. Especially in complex assembly tasks, this aggravates the assembler from understanding the assembly task [17]. In addition, more and more unskilled employees and employees with little language skills are working in companies, which leads to errors due to, for example, incorrect reading or incorrect interpretation of work documents [22].

One approach to improve the comprehensibility of work instructions and assembly instructions can be innovative ways of knowledge transfer with augmented reality (AR) and virtual reality (VR) [24]. For example, AR technology can be used to respond to rapidly changing product variants with dynamically adaptable assembly instructions [21]. The term augmented reality describes the addition or extension of virtual content to reality [9], which is registered in real time, interactively and spatially correctly registered [2]. Besides the presentation of virtual content on a transparent screen of a head-mounted display (optical see-through) [25], screens can be used to augment a camera video with content (video see-through) [7]. In combination with the optical tracking method of marker tracking, simple AR applications can be created that can be used, for example, with a smartphone or a tablet. In addition to black and white patterns, any other images can be used as markers, which are recognized and used by the camera of the device to superimpose virtual content in the correct position in relation to the respective marker [25]. Since, in addition to a camera for capturing the markers, only mostly free software is required for programming, marker-based AR represents a low-cost option for the use of AR [25]. In addition, markers are simple to use and easy to learn even for non-experts. The markers can be quickly created, printed out and then freely placed [9].

The use of optical and video see-through AR technology to provide assembly information in industry has been investigated in various studies. Wiedenmaier et al. investigate how assembly can be supported with AR and compare AR instructions with classic paper instructions for various complex assembly tasks [27]. The study considers the assembly of a car door as an example of a common activity in small- to medium-sized series production in the automotive industry. The results show that AR instructions are more suitable than conventional paper instructions, especially for complex assembly tasks [27]. The study by Funk et al. investigates the influence of AR instructions on individuals with different prior knowledge about the assembly task [11]. The developed AR system supports the assembler, for example, by marking the correct part location for the currently required component or by automatically indicating assembly errors [11]. The study considers the assembly of a motor starter over the period of eleven days with experts in assembly at this workplace and with unskilled workers. The results show that the experts assembled significantly slower with AR support than without. In contrast, the unskilled workers were significantly faster with AR support than without [11]. In Blattgerste et al.’s study, a video see-through application for smartphones is implemented in addition to an optical see-through application for data glasses [5]. A Lego Duplo assembly was chosen as the assembly task, with each assembly step consisting of selecting the correct brick and assembling it in the correct location. Subjects who assembled using the paper instructions were significantly faster in brick selection than subjects using the smartphone AR application. There were no significant differences in the number of assembly errors between the different types of instructions [5]. The use of AR markers in combination with video see-through AR application in assembly instructions has also been investigated in other studies. Hou and Wang found that subjects with AR assembly instructions assembled a Lego assembly faster and with fewer errors than with printed paper instructions [16]. Loch et al. also studied the assembly of a Lego assembly, but they compared video instructions with AR instructions. The study with 17 subjects showed no significant difference in assembly time and strain (NASA-TLX), but significantly fewer errors were made in the AR group [20]. Since both groups built different Lego models in the study, comparability of the complexity of the assembly task is questionable.

Instead of a Lego assembly, Hořejší considered the assembly of a siphon and also compared assembly instructions in paper form with AR instructions. In addition to a 3D model of the current component, the AR instructions also showed the correct parts stores with arrows where the required component could be found. The study showed a large time saving in the assembly of the first assembly pass [15]. However, this could be related to the lack of support in the part selection from the parts stores of the paper manual.

Some of the assembly tasks in the presented studies differ greatly from actual assembly in companies. In order to ensure that the laboratory studies can be transferred to a real assembly task in companies, the following section examines the comparative evaluation of paper instructions and AR instructions at a real assembly workplace for the assembly of emergency handles for unlocking doors in rail vehicles. The focus is on the influence of the type of instruction on the assembly time, the errors made, the usability and the strain on the employees. The following hypotheses are tested in the study: The instruction type affects the assembly time (H1) and the number of assembly errors (H2). The AR instruction differs in usability from the classical paper instruction (H3) and the instruction type affects the strain of the employees (H4).

The special feature of the study lies in the real application case, since this is carried out at the assembly site of an industrial company with a real assembly. In contrast to frequently used student samples, the study is conducted with trainees of the company, who are all undergoing technical training and are familiar with manual assembly. This means that the test subjects correspond to the target group for which the possible use of AR instructions in assembly is interesting.

2 Method

2.1 Assembly Task

The assembly task for the study is the manual assembly of an emergency handle for opening railroad doors. The emergency handle is manufactured in up to 30 different variants. The emergency handle used in the study consists of 16 components, and the assembly is divided into various activities, such as positioning, screwing, pressing and aligning components. Figure 1 shows a model of the assembly and a fully assembled emergency handle.

Figure 1

3D model of the emergency handle to be assembled (left); assembled emergency handle (right).

Figure 2

Assembly workstation part containers, assembly jigs and a press-fit jig (left); tools required for assembly (right): combination wrench size 8 (1), hexagon socket wrench size 4 (2), needle-nose pliers (3), plastic hammer (4).

The handle is assembled piece by piece at a single workstation. Despite the complex task, the assembly can be carried out independently by unskilled persons after a short training period. The workstation with an assembly table (90 cm × 150 cm) has parts stores which are marked with item numbers. Assembly fixtures are available as assembly aids, into which components can be inserted to facilitate correct positioning. In addition to simple hand tools, a press-in device is available for pressing in metal pins. The workstation and the tools required can be seen in Figure 2.

2.2 AR Assembly Instruction

For the study, marker-based AR assembly instructions for the assembly of the emergency handle were created as a smartphone application. The basis for this was an existing paper manual used in the company. The contents of the paper instructions were transferred to the AR application. For the AR application, the assembly process was divided into seven steps, each of which was indicated with a marker. These markers are recognized by the application and used to display suitable information in the correct position. In each of the seven steps, the employee is shown 3D models of the components currently required, a text field with the corresponding item numbers of the parts stores, and, if necessary, further information about the work step to be performed. In some cases, the steps consist of several assembly sub-steps that build on each other. For this purpose, the application was extended by two buttons for switching the displayed information forward and backward. The use of the application and the display off the smartphone can be seen in Figure 3.

Figure 3

Smartphone view for assembly step one, sub-step one of two, with indication of the part item numbers as well as the work request “Operate press”, buttons “Back” and “Next (1/2)” for independent browsing of information (left); display of the correctly assembled parts (right).

The AR application was created in the 3D development environment Unity (version 2019.1.12f1) and the Vuforia Kit for marker-based AR. In Unity, the 3D models of the assembly were linked to the markers of the corresponding steps and partially supplemented with text for the position number and other notes.

2.3 Measurement

Technology affinity was assessed using the standardized TA-EG questionnaire with 19 items on a five-point Likert scale from “Strongly disagree” to “Strongly agree” [19]. The items are statements about the attitude towards and use of electronic devices, such as “Electronic devices make people independent.” or “Electronic devices cause stress.”. After inverting the values of negatively worded items, the mean values of the four scales enthusiasm, competence, negative and positive attitude are averaged to an overall value. The values range from 1 (very low technology affinity) to 5 (very high technology affinity).

Execution times and the number of assembly errors were evaluated by video recording for each of the five assembly runs.

The System Usability Scale (SUS) questionnaire was used to measure usability [6]. The questionnaire consists of ten items, five positively and five negatively worded. The items are to be answered on a five-point Likert scale from “I strongly disagree” to “I strongly agree”. According to Brooke’s specifications, the negatively formulated items are inverted, then all values from 1–5 are converted to 0–4, summed and multiplied by a factor of 2.5 [6]. This results in a possible total value between 0 (very bad usability) and 100 (very good usability).

Figure 4

Subjects during assembly of the emergency handle with paper instructions (left) and AR instructions (right).

The strain was surveyed with the NASA-TLX questionnaire [13]. This consists of subjective ratings of the six sub-items of mental, physical and time demands, as well as performance, effort, and frustration on a 20-point scale ranging from low to high. In pairwise comparisons, the participant selects from each two of the six sub-items what was more significant for the work task. This allows the overall value to be weighted. The total value lies between 0 (no strain) and 100 (extreme strain).

3 Results

At the beginning of the evaluation, all scales were tested for normal distribution with the Shapiro-Wilk test and for homogeneity of variance with the Levene test. Except for the number of errors made, all values on the scales presented below show a normal distribution and homogeneity of variance.

Hypothesis H1 tests whether the assembly completion time differs between the paper and AR groups. The assembly times determined from the videos can be seen in Figure 5.

Figure 5

Boxplots of the assembly times of the paper group (gray) and AR group (blue) grouped according to the assembly passes.

To test H1, a mixed ANOVA was used. The Mauchly test showed violations of sphericity in the assembly times, therefor the Greenhouse–Geisser adjustment was used. In addition to Levene’s test for homogeneity of variances, the homogeneity of covariances was assessed by Box’s test.

There was a statistically significant interaction between the assembly runs and the group, Greenhouse–Geisser F(2.02, 28.27) = 7.82, p = .002, partial η² = .36. After a significant interaction was found, t-tests were used to test the main effects of the between-subjects factor (group). Table 1 shows the means, standard deviations and results of the t-tests.

Then the main effects of the within-subject factor (assembly run) on the assembly time was tested in groups with repeated measures ANOVAs and Tukey post-hoc tests. Results show a statistically significant effect of the assembly run on the assembly times in both the paper and AR group, Greenhouse-Geisser F(2.55, 17.83) = 93.80, p < .001, partial η² = .93 (paper group); F(1.56, 10.92) = 129,49, p < .001, partial η² = .95 (AR group). The results of the post-hoc tests are shown in Table 2 for the paper group and in Table 3 for the AR group, respectively.

Figure 6

Average assembly errors of the paper (gray) and AR (blue) group over the assembly runs.

Hypothesis H2 examines the number of errors made during assembly for the two types of instruction. For this purpose, the video recordings were evaluated and the errors made per person and assembly run were considered. The paper group made an overall average of 3.25 errors (SD = 2.49 errors) and the AR group 3.12 errors (SD = 1.13 errors). Figure 6 shows the average number of errors made per assembly run by group. Since ANOVAs are relatively robust to violations of the normal distribution, H2 was also tested with a mixed ANOVA. The Mauchly test showed violations of sphericity in the assembly times, therefor the Greenhouse–Geisser adjustment was used.

There was a no statistically significant interaction between the number of errors and the group, Greenhouse–Geisser F(2.36, 32.99) = .53, p = .63, partial η² = .04.

Hypotheses H3 and H4 test whether usability (H3) and strain (H4) are different when using the AR application compared to the paper manual. The two results are given in Figure 7.

Figure 7

Usability (SUS) (left) and the strain (NASA-TLX) (right).

To test for differences, a two-tailed t-test for independent samples was calculated in each case. The average usability score obtained with the SUS is higher in the paper group (M = 89.06, SD = 5.16) than in the AR group (M = 84.06, SD = 7.31). However, this difference is not significant (t(14) = −1.58, p = 0.137).

The average strain measured with the NASA-TLX is lower in the AR group (M = 35.41, SD = 14.71) than in the paper group (M = 37.42, SD = 13.59). However, this difference is also not significant (t(14) = −0.28, p = 0.780).

After completing the assembly task, the test subjects were interviewed about their impressions of the previously used instruction method. The subjects’ statements were recorded in bullet points and then classified into positive and negative aspects of the instruction method in a table.

In the AR group, the fact that the smartphone occupies one hand during use was mentioned as annoying. It is also difficult to place the smartphone on the assembly station. The use of the smartphone in assembly was described as taking some getting used to. Further statements concern the technical challenges of marker-based AR applications. Here, the sometimes longer times for recognizing a marker were mentioned. It was criticized that displayed content disappears as soon as the smartphone is put aside for assembly. In addition to the virtual 3D models of the components, this primarily concerns the indication of position numbers of the parts stores. After the smartphone is picked up again, the markers had to be recognized again in order to call up the content again.

The three-dimensional representation of the components in particular was mentioned as positive and very helpful. Here, the positionally correct display of the 3D models on the assembly table was seen as an advantage. The AR visualization with three-dimensional models was preferred to the otherwise usual 2D illustrations of models or photos of components. The possibility of viewing the models of the components from different sides through different camera positions was also mentioned positively. The design of the AR instructions was described as clear, the small amount of text in combination with many 3D models was perceived as good. Further positive feedback was received for the fact that only the required information was displayed for each work step. Table 4 summarizes the statements.

4 Discussion and Conclusions

The study shows mixed results of assembly times. In passes one and two, the paper group’s subjects are significantly faster than those of the AR group. This result differs from the findings of Hořejší, where especially in the first assembly run a high time saving could be achieved with the AR instruction [15]. This difference could be due to the fact that in the present study, an attempt was made to keep the contents of both instructions identical, whereas Hořejší gave the AR group additional support in part selection from the parts stores. After no differences in times for runs three and four, the AR group is significantly faster than the paper group in run five. It can be assumed that the initially good times of the paper group in comparison to the AR group, despite the AR group practicing with a training AR application, stem from the subjects’ prior experience in using the paper instructions. The AR group, on the other hand, still had to become familiar with the new type of instructions and how to handle them during assembly. The progression of times and the faster assembly in run five nevertheless point to the general potential of AR instructions.

The results do not allow a clear statement regarding errors. The two groups do not differ either in individual runs or in the consideration of the total number of errors. Both groups make very few errors, whereby the errors made are often due to carelessness such as forgetting to assemble small components or checking fine details (e. g. correct alignment of the shafts).

The usability of both manuals, with an average value of 89.06 (paper group) and 84.06 (AR group), is just above or below a rating of excellent (at 85) [3]. Thus, both are in a very good range; differences in usability could not be demonstrated. The subjective strain of the employees due to the type of instruction is also on a similar level with an average of 37.42 (paper group) and 35.41 (AR group) and does not show any significant differences. In order to classify the values, results of other studies are referred to. In a meta-analysis, the NASA TLX values of 237 scientific studies with a total of 1173 data sets were evaluated. The median of all values is 49.93, 25 % of the values are below 36.77 [12]. Thus, the present values can both be assigned to a rather low strain.

Although the experimental documents deliberately omitted information about the type of instruction of the other experimental group, it cannot be ruled out that subjects knew about it. Possible bias effects due to higher motivation of the AR group are therefore possible.

The results show partial overlaps, but also differences to similar studies with student samples and artificial assembly tasks. Here, the importance of special samples becomes clear in order to enable the transferability of results. The comparable high usability and low strain make marker-based AR applications interesting for manual assembly, especially due to the easy entry and low cost. Although no positive effects on assembly times and errors could be demonstrated in this study, further research can help to understand the influence of AR on assembly more precisely and to apply it in a targeted manner.