1 Introduction

Augmented reality technology has the capability to superimpose virtual information to the real world. Augmented reality (AR) has been applied in different fields including manufacturing, medical, interior design and education in order to increase the effectiveness of the current work in these areas (Che Hashim et al. 2018; Groover 2002; Hossain et al. 2016; Muhammad Nizam et al. 2018; Rahman et al. 2017). The recent growth of headset cases such as Samsung Gear VR, Google Cardboard and Zeiss VR One for smartphone devices can all be used to generate a stereoscopic vision with a larger field of vision. It is one of the categories in mobile augmented reality (MAR). Given the growth of smartphone devices, the use of MAR platforms and the availability of headsets are quickly converting smartphones into a powerful platform where the costs are relatively cheap, and the devices are easy to acquire compared to dedicated wearable AR devices such as Magic Leap and Hololens. Aside from that, smartphone-based AR is suitable for performing various kinds of tasks given the features such as portability, ready access and 3D stereoscopic display which provides the user with depth judgment and a broader field of vision offering a better visual experience.

AR on mobile devices also encourage knowledge transfer to occur more efficiently, which has been viewed as a research topic by educators (Klopfer and Squire 2008). AR is a convenient tool to deliver hands-on training which can display virtual instructions in the context of real-world objects to the user. Likewise, AR can be applied to an assembly line and operation, in fulfilling the same role as paper instruction manuals and other instructional methods to guide users through an assembly/disassembly sequence (Baird and Barfield 1999). AR also aids memorisation by leveraging the association of the presented information with a physical location in the sense of a real-world space (Biocca et al. 2001; Neumann and Majoros 1998; Tang et al. 2003). In fact, many researchers conclude that AR can also help to motivate users (Blagg 2009; Di Serio et al. 2013; Martín et al. 2012; Radkowski et al. 2015; Tarng and Ou 2012). When users are motivated, they tend to engage more in AR-based learning material and gain more exposure to the learning content; thus, more knowledge is retained. Also, the training time for new employees is shortened through improved memorisation using AR. By applying AR, indirectly, costs are also reduced since employees can adapt quicker, thus contributing more efficiently towards production operations (i.e. assembly/disassembly) with less training expenses.

Few studies investigated knowledge retention using AR technology such as Rios et al. (2011), where they made a comparison in transferring knowledge between AR training, traditional teaching techniques and audio–visual training tools. In a separate study, Westerfield et al. (2015) performed research on the assembly of computer motherboards with and without intelligent support by employing AR technology and recorded the result concerning knowledge retention. In another study, Hou and Wang (2013) used LEGO MINDSTORMS as an assembly testing bed to analyse the gender factor, and Huang et al. (2019) explored knowledge retention relating to science information using virtual reality and AR. In their study, knowledge retention was examined using a pre-test and post-test experiment design by participants answering knowledge-related questions. The experiment was performed to determine how much knowledge could be retained by the participants. However, in the mentioned research the post-test was performed immediately after the pre-test phase.

In contrast to the previous research, the aim of this study is more interested in examining knowledge retention in long-term memory (LTM) using stereoscopic vision using MAR platform. The MAR platform was also evaluated. LTM takes several hours to stabilise, and many of the studies in medical-related fields practice the test in long-term memory after the initial training of 24 h or more (Cammarota et al. 2007; Nader and Hardt 2009). Hence, the experimental design to address the gap between the short periods between the pre-test and post-test extended (more than 48 h) to ensure the skill or knowledge gained by the participant from the pre-test phase is stored in long-term memory. The developed system focused on a disassembly process in which the user detached components either by hand or with the assistance of common handheld tools. Such processes involved preparing the proper tools in performing such tasks and unfastening multiple components to reach a specific component or vice versa.

2 Related work

Previous early studies have revealed that an AR-type training system has a certain advantage over traditional training methods on the knowledge retention. Rios et al. (2011) conducted a comparison of transferring knowledge between AR training, traditional teaching techniques and audio–visual training tools. However, only the AR method was evaluated, and the result was normalised to compare with the results from prior research. Notably, the consistency of the experiment is also unclear making it difficult to judge whether it was a valid comparison. A group of researchers from the Embry-Riddle Aeronautical University investigated AR technology for training purposes of a turbine engine aircraft oil pumps. The AR system provided virtual text information regarding the parts within the oil pumps (Macchiarella et al. 2005). Several experiments that were performed revealed that AR technology is capable of helping participants to retain knowledge in the long-term memory compared to video and paper type of training method which degrades from the memory faster (Macchiarella et al. 2005; Macchiarella and Vincenzi 2004). A desktop-type AR was used in such experiments. The benefits of desktop-type AR technology that aids in long-term memory retention also include topic like language learning (Solak and Cakir 2016) together with digestive and circulatory systems (Pérez-López and Contero 2013). Having said that, Solak and Cakir (2016) provided an attractive animation related to vocabulary and pronunciation. Hence, based on the results, participants were able to recall the learned information better in AR setting compared to the traditional methods where the vocabulary was written on the board. In addition, Pérez-López and Contero (2013) created an interactive AR for fourth-grade students’ on digestive and circulatory systems. Using this AR, students were able to bring the AR marker closer or away from the camera to visualise different parts of the human body. The assessment also revealed that AR aids in knowledge retention better than the traditional methods after 2 and 4 weeks of study duration. On the other hand, Noll et al. (2017) applied MAR to diagnose various skin conditions by placing the marker to visualise the skin condition images of the users. They examined the participants using MAR and mobile application (without AR functionality). Both MAR and mobile application groups gained equivalent knowledge following the pre-test, but MAR demonstrated better long-term retention of knowledge. However, not all previous studies always showed the benefits of AR, as AR failed to demonstrate the advantages in knowledge retention compared to computer-based training and traditional lecture in retinopathy-based diagnosis (Bergeron et al. 2019).

In reference to long-term evaluations, a minimum of two phases (pre-test and post-test) is required where the post-test should be conducted at least 24 h after the pre-test (Cammarota et al. 2007; Nader and Hardt 2009). However, it is not a case for evaluation undertaken by previous studies (Gavish et al. 2015; Gonzalez-Franco et al. 2017; Hou and Wang 2013; Huang et al. 2019; Westerfield et al. 2015). Gavish et al. (2015) conducted an evaluation study for AR, VR and instructional video applied to industrial maintenance and assembly task training, and the research results were encouraging by using AR as the training method. Animated AR systems are quite attracting in representing the movement of an assembly step. Here, the positive effect of cognitive facilitation when using an animated AR system was confirmed by Hou et al. (2013), while Westerfield et al. (2015) developed an AR intelligent tutoring system using a head-mounted display connected to a computer in the assembly of a computer motherboard. Here, they performed an evaluation between the AR system with intelligent support and without intelligent support. Results showed participants performed faster and recalled more knowledge with the intelligent support AR system. On the other hand, Gonzalez-Franco et al. (2017) compared AR and face-to-face training with respect to conduct maintenance on an aircraft door. The results indicated that performance levels for both training types were not significant. However, both methods enabled the participants to develop a more hands-on memory of the procedure than that of abstract knowledge. Indeed, AR could be useful for remote training when face-to-face training is not an option. The experiment design by the researchers mentioned above was undertaken two phases: the pre-test and post-test. Data were collected that comprised of task completion time, the number of errors, a knowledge test, and a subjective evaluation questionnaire was utilised to determine the performance of the training method. However, evaluation of the experiment is questionable, especially the evaluation with respect to memory because they commenced the knowledge test straight after the participants completing the pre-test phase. This is similar to the work of Huang et al. (2019), where the knowledge test concerning the solar system was performed straight after the pre-test phase. Accordingly, research gaps are evident and remain from the research mentioned above, especially on the design of experiment (DoE) with the post-test conducted following the pre-test. On the other hand, stereoscopic-based mobile augmented reality was investigated in this study, different from the previous investigations which utilised the desktop-based approach (Hou et al. 2013; Macchiarella et al. 2005; Macchiarella and Vincenzi 2004; Pérez-López and Contero 2013; Solak and Cakir 2016) or mobile augmented reality (Huang et al. 2019; Noll et al. 2017). Nevertheless, different experiment design or evaluation methods concerning knowledge retention can be undertaken to confirm the benefit of AR in knowledge retention. According to Cammarota et al. (2007), Nader and Hardt (2009), short-term memory develops within a few seconds or minutes after performing an action and may last for several hours. However, the consolidation of long-term memory proceeds gradually and can last as long as 24 h or more. Therefore, this study designed an experiment in which knowledge surrounding the training was evaluated after 48 h to ensure the knowledge is retained in long-term memory.

Notwithstanding, the study by Holliman (2005) presented relevant information regarding 3D display technology, mentioning that most perceptual cues that humans use to visualise 3D structures are available in 2D projections. As such, people can make sense of photographs and images displayed on a television screen, at the cinema, or on a computer monitor, whereas binocular vision offers humans the benefit of depth perception resulting from the small differences in the location of homologous or corresponding points in visualising two images via the retina of the eyes. This is known as stereopsis (literally solid seeing) and provides precise data on the depth relationships of objects in a scene. Indeed, 3D stereo vision has several advantages over monocular vision such as correct depth measurement, spatial localisation, recognising disguised objects and physical surface awareness. Importantly, 3D visualisation techniques can also help to improve user memory (Czerwinski et al. 1999). As such, it appears that AR technology can utilise stereoscopic vision to present better visual feedback for users to aid in their learning. Hence, this study also aims to determine the effect of stereoscopic vision in long-term memory using a MAR platform.

3 MAR system framework

The effect of integrating MAR technology with stereoscopic technology on the knowledge retention of users is investigated in this study. A disassembly training case was chosen for this experiment, given its natural action in supporting users concerning cognitive and psychomotor aspects in performing the tasks. A PlayStation 3 was selected in this study given the disassembly process included various types of actions such as picking up and handling screwing tools, performing screwing actions, removing cantilever snap-fit, lifting-up the metal lever, etc. More importantly, the participants would not be able to disassemble the PlayStation 3′s components without any assistance, guidance or using a manual. It will require some guidance to proper disassemble the parts. This section discusses the development of a suitable framework in which a stereoscopic-based MAR application was developed describing the disassembly process for a PlayStation 3 console.

3.1 Stereoscopic-based MAR application framework

In developing an AR system, multiple elements need to be integrated in order to create a robust AR system. An extensive literature review was performed to study previous AR systems concerning their processing unit, tracking method, display technology, display type, information visualisation type and interaction technique, as summarised in Table 1. With the finding in Table 1, it will help the novice developers to understand the necessary elements in AR application development. By adopting the conceptual framework from the research work of Sadik and Chun (2017) and the elements of AR, as shown in Table 1, a stereoscopic-based MAR framework for the disassembly case specific to this study was designed. Figure 1 shows the designed framework consisting of the physical environment and the AR environment. The physical environment consists of the devices, while the AR environment consists of certain elements selected from Table 1.

Table 1 Elements of the AR system
Full size table
Fig. 1
figure 1

Stereoscopic-based MAR system framework

Full size image

3.2 Physical environment

The physical environment consisted of an AR smartphone device in which the processing power of the device needed to be considered. In previous research, a device having minimum computational power (2 GB RAM and 1.4 GHz CPU) was used. Such computational power allows lots of mobile devices using AR technology. On the other hand, most devices tend to suffer from overheating, lag, hick-ups and other problems due to low computational power. Therefore, considering these issues, a Samsung Note5 device was chosen for this experiment having high computational power (CPU: Octa-core (4 × 2.1 GHz Cortex-A57 & 4 × 1.5 GHz Cortex-A53), and with 4 GB of RAM. In 2016, this device was considered as one of the highest computational powered mobile phones available.

The second device that was used in the experiment was a headset case that provided a stereoscopic view. The headset case can be described as a physical case without any processing unit such as a simple VR box, Samsung Gear VR, mi VR Play, etc., and is designed to be used with a combination of smartphone devices. Samsung Gear VR Innovator Edition was selected for this research, given its mobility and compatibility with the Samsung Note5 device. Also, it can provide a stereoscopic view along with other useful features, namely is lightweight, has a 96-degree Field of View (FOV), and video-see-through via the attached device camera. The advantage of having a larger FOV can help to increase performance, such as accuracy and faster completion time related to physical tasks (Toet et al. 2007). The processing power, FOV, etc., in Samsung Note 5 ensure the compatibility with Vuforia which it is the AR development platform used in this study. This was one of the main reasons for choosing the Samsung Gear VR Innovator Edition.

3.3 AR environment

Creating an application for a specific assembly/disassembly task requires preparing several elements before beginning the development process, especially regarding the AR environment part. Vuforia was used in this study to develop the AR application as it supports stereo rendering via a video-see-through device. The configurations such as the Camera Offset that correspond to the interpupillary distance were adjusted to ensure the quality of stereo rendering (Vuforia 2016). Even though the stereo vision was roughly at 80 degrees following rendering, it was appropriate for this study given users still felt comfortable in performing the task while seated. Aside from that, the required AR elements included the virtual content (3D models and 3D model animations), tracking target, and interaction technique.

3.3.1 3D models and animations

According to Chang et al. (2017), product disassembly sequence information is the key element required before developing an AR application (i.e. disassembly). The AR application should comprise of virtual content concerning the case/object that the user will be working on. Radkowski et al. (2015) mentioned that manual assembly is a process that involves using various tasks such as alignment and fastening for multiple pieces or parts in creating the final product. The disassembly process is the contrary task of the assembly process which can be defined as the process of the systematic removal of desirable constituent parts from an assembly while ensuring there is no impairment of the parts during the process (Brennan et al. 1994). The content of the disassembly process can be presented in different forms, such as an image, audio, video and animation. In this study, the disassembly sequence was prepared and presented in a 3D model with animation form in the AR environment as one of the elements in the framework. This research utilised PlayStation 3 console 3D model components, and 3D animations which visualised and animated the disassembly steps of a PlayStation 3 console. The animation included the process of removing the cover, CD-ROM, power supply, hard drive and the Wi-Fi-card. The developed application disassembly task sequence was derived from the PlayStation 3 console disassembly manual.

3.3.2 Target tracking

The AR application requires several methods to track the AR marker to display the virtual objects in the physical environment correctly. Many methods can be employed for such purposes and that address three categories: images, environment and objects. Different situations also require different tracking methods as one may not suit the others. For example, 3D object tracking does not require an additional maker on the target which the target/product can maintain its existing appearance without any modification. However, 3D object tracking is challenging as it may suffer from issues such as stability tracking and overheating. Hence, the developed AR application in this research used an image of the PlayStation 3 console as an AR marker (Fig. 2) to augment the virtual objects given the reliability and efficiency (Yee et al. 2015; Zheng 2015). It received a 4 out of 5 stars quality rating in the Vuforia marker’s standard; hence, the tracking is robust.

Fig. 2
figure 2

AR marker with PLAYSTATION 3 image

Full size image

3.3.3 Interaction technique

The interaction technique is employed for users to interact with the virtual content in the AR environment. There are different ways to interact with augmented objects such as through vision, gaze, voice commands, haptic, gestures, and remote-control interaction (Arshad et al. 2016; Lam et al. 2015; Wook et al. 2016). The developed application in this study used head movement tracking (see Fig. 3) as an interaction technique, which facilitates user interaction with the sequence of the disassembly 3D animation via 3D buttons. Here, the users are only required to move their head to control a pointer and point to a 3D button in order to trigger the disassembly animation. The users were not required to hover the cursor on the button for a certain amount of time to trigger the animation. It provides a hands-free interaction experience. As such, users can interact with the AR environment without any difficulties of switching/controlling with additional devices while performing the task with their both hands.

Fig. 3
figure 3

Using head movement interaction to trigger a disassembly process: a, b user controls the pointer via head movement to select the 3D button; c, d animation of using a screwdriver to unfasten the screw(s), and e, f animation of removing the cover. (The original rendering output was in a stereoscopic view and was consequently cropped into a single image to provide clearer labelling and visualisation for this figure)

Full size image

4 MAR experiment design

Figure 4 illustrates the design of the experiment, consisting of two main phases: the pre-test and post-test. An instructor accompanied and instructed the participants during the evaluation. The participants were initially divided into two groups: the MAR group and paper-based group. In the pre-test phase, the participants were requested to provide background information and answer questions related to their existing experience concerning AR with respect to performing a disassembly task. The experiment was subjected to only those participants who had no prior knowledge or having a low level of knowledge regarding the task to be performed. A knowledge test was also carried out to check their level of knowledge regarding the device (PS3). Participants were excluded from participating in the experiment if the results showed they were familiar with the task or the AR application. This was followed by briefing the participants on the disassembly task in which they needed to solve according to the assigned assistance tools (Fig. 5) which they would be using consecutively for three times. The participants could take a rest while waiting for the instructor to reassemble the parts. The instructor also recorded the task completion time for each participant.

Fig. 4
figure 4

Experimental procedure

Full size image
Fig. 5
figure 5

Participant disassembling the PlayStation 3 console using: a MAR system, b paper-based manual

Full size image

The post-test was conducted after 48 h following the pre-test. Here, the participants were asked to complete the same knowledge test, redoing the same task without any assistance of using the tools. This stage determined how much information the participants could retain in recalling the disassembly steps from the first experiment in which they completed with the assistance of using the tool to complete the task. The time was also recorded, and the instructor observed and recorded how many times the participants made a wrong move. For example, when the participants tried to remove one of the console components, it was completely released from all the attached screws.

Before the participants were requested to complete the usability questionnaire, they were instructed to perform the task with another group’s assistance tool in the pre-test phase for them to experience both assistance tools. A questionnaire was provided to participants to complete at the end of this experiment in order to evaluate the assistance tool concerning its usefulness, ease of use and satisfaction. A preference question was also presented to gather the participants’ opinions regarding which method they preferred.

4.1 Experiment task

The designed experiment as mentioned earlier was disassembling a PlayStation 3 console which consisted of six tasks in which participants were required to remove numerous screws, disconnect the cable and remove the part(s) employing correct movement during the procedure. The participants were asked to remove the first cover, second cover, CD-ROM, power supply, hard drive and the Wi-Fi-card from the console with or without the MAR tool. Figure 3 illustrates the steps to remove the second cover from the console, while Fig. 5 shows participant was performing the experiment using the MAR and paper-based manual.

4.2 Experimental measurements and variables

The independent variables for this experiment were the tools employed for the training, which included the MAR application and the paper-based manual. The measurements adopted in the experiment included the time consumed and the number of disassembly sequential errors in completing a task. The score for the knowledge test was recorded, which comprised of questions that the participants were asked to identify the components based on the image of components of the PlayStation 3 console. The collected data were compared between the testing phase: pre-test (MAR) vs pre-test (Paper-based); post-test (MAR) vs post-test (Paper-based); and between the testing phases: pre-test (MAR) vs post-test (Paper-based).

4.3 Participants background information

Thirty university students were recruited to participate in the study comprising of 25 males and five females. The age of participants ranged between 21 and 32 years, with the average age being 26.94 years. The participants were not paid for their involvement in the experiment. All participants were university students having diverse educational backgrounds; 23% were pursuing a degree, 60% were pursuing a master’s degree and 17% were pursuing a PhD. Figure 6 displays the background knowledge of the participants related to this study. Sixty per cent of the participants were familiar with the assembly task using the traditional paper-based manual, and 30% of participants had previously experienced AR technology. However, none of the participants had any prior knowledge or experience in any assembly-related tasks applying AR technology. Feedback from the participants indicated that around 60% had prior knowledge of computer hardware.

Fig. 6
figure 6

Background knowledge descriptive statistics

Full size image

The descriptive statistics revealed that 93% of participants had no prior knowledge whatsoever regarding any of the PlayStation 3 console parts, while 7% had only general knowledge, also claiming they had not assembled/disassembled a PlayStation 3 console previously. As such, the results obtained from these questions assured the researcher that all participants did not have sufficient knowledge in addressing all aspects of the experiment and, therefore, considered suitable to participate in the experiment. Based on Q1 from the background information section that involved the 18 participants who had experience in the assembly/disassembly task using a paper-based manual before, we divided the participants equally into the MAR group and paper-based group; 9 participants with experience and 6 participants without experience in each group.

Background knowledge of the participants gathered by asking the following questions:

  • Q1 Have you ever performed an assembly/disassembly task using a paper-based manual before?

  • Q2 Have you ever experienced augmented reality technology?

  • Q3 Have you ever experienced augmented reality in an assembly/disassembly task?

  • Q4 Are you familiar with computer hardware setup?

  • Q5 Do you have any knowledge about PlayStation 3 Console parts?

5 Experimental results

This section discusses the evaluation process and results collected from the experiment to develop the MAR application. The main goal of the evaluation process was to determine whether Stereoscopic-based MAR affects human knowledge retention. The collected data were analysed using statistical tests based on the design of experiment (DoE). As mentioned earlier, thirty participants were recruited and divided into two groups. As such, the assumption regarding the distribution of the sample was not normal due to the small sample size and Shapiro–Wilk test’s confirmation with p < 0.05. Consequently, the Mann–Whitney U test and Wilcoxon test were used to analyse the collected data (Marusteri and Bacarea 2010). The Mann–Whitney U test was used to determine the significance of the tool used to assist the participants of the two groups participants in the pre-test or post-test phases, while the Wilcoxon test was used to determine the significance of the tool to assist participants within the same group of participants between the pre-test phase and the post-test phase.

5.1 Knowledge test results

The knowledge test was performed in both the pre-test and post-test. The pre-test phase focused on obtaining existing knowledge about the PlayStation 3 console device, while the post-test focused on how much knowledge the participants had gained or remembered after performing the task. The knowledge test consisted of six questions, each question representing a part of the device that the participants had to disassemble and identify its name.

5.1.1 Results of the knowledge test in the pre-test phase between the groups

The knowledge test results for the pre-test phase are shown in Table 2. The results showed that all MAR group participants could not identify the PlayStation 3 console’s first cover, second cover and power supply. For the CD-ROM and Wi-Fi-card, only one out of fifteen participants answered it correctly, whereas, for the hard drive, seven participants out of fifteen answered it correctly. The knowledge test in the pre-test phase results for the paper-based group showed that one out of fifteen participants could identify the first cover and power supply for the PlayStation 3 console, but none could identify the second cover and the Wi-Fi-card. Only two participants out of fifteen could identify the CD-ROM, and seven participants out of fifteen could identify the hard drive. The correct answers overall for the MAR group were nine, and eleven correct answers for the paper-based group, while the number of incorrect answers was eighty-one for the MAR group and seventy-nine for the paper-based group.

Table 2 Knowledge test statistics in the pre-test phase
Full size table

The results showed that the percentage for identifying the hard drive for both groups was higher compared to identifying the component parts. This is because the hard drive used in the PlayStation 3 console is similar to a hard drive used in a standard [desktop] computer. As such, this result confirmed that the participants had little knowledge regarding the PlayStation 3 console based on the collected data obtained from the background knowledge test. By providing the one score for the correct answer, the MAR group scored an average of 1.00 (SD = 1.23), and the paper-based group scored an average of 1.22 (SD = 1.60) out of 10 total scores. In addition, Fig. 7 illustrates the mean and dispersion of knowledge test data using a box plot. The Mann–Whitney U test used to analyse the collected data showed no significance between the two groups due to the high p value (U = 108.0, p = 0.87). Accordingly, this indicates that both groups had equal knowledge concerning the PlayStation 3 console, which is deemed suitable for this study.

Fig. 7
figure 7

Box plot with mean scores from the knowledge test in the pre-test and post-test phases

Full size image

5.1.2 Results of the knowledge test in the post-test phase between the groups

The results of the knowledge test in the post-test as shown in Table 3 indicated that fourteen out of fifteen participants in the MAR group were able to identify for the first cover, CD-ROM and hard drive of the PlayStation 3 console correctly, whereas only thirteen out of fifteen participants could identify the power supply and Wi-Fi-card correctly. In addition, twelve out of fifteen participants identified the second cover correctly. In the paper-based group, six out of fifteen participants could correctly identify the CD-ROM, twelve out of fifteen participants could identify the Wi-Fi-Card correctly, and thirteen out of fifteen could identify the first cover and hard drive. In addition, five out of fifteen participants in this group could identify the power supply along with eight out of fifteen participants identifying the second cover. The overall correct answers for the AR group were eighty and fifty-seven for the paper-based group, while the number of incorrect answers was ten for the MAR group and thirty-three for the paper-based group.

Table 3 Knowledge test statistics in the post-test phase
Full size table

By providing the one score for the right answers, the MAR group scored an average of 8.89 (SD = 1.36), and the paper-based group scored an average of 6.33 (SD = 1.80) out of 10 total scores per participant. Figure 7 illustrates the mean and dispersion of knowledge test data using a box plot. The collected data were then analysed employing the Mann–Whitney U test, indicating that the results for the two groups were significant due to the low p value (U = 31.5, p < 0.001). Therefore, based on the result, the MAR group had gained much better knowledge compared to the paper-based group in the post-test phase. The MAR group had 40% more correct answers compared to the paper-based group.

5.1.3 Comparison of the knowledge test between the pre-test and post-test phase within the groups

A Wilcoxon test was performed in order to identify the significance of the gained knowledge in the same group of participants between the pre-test and post-test. The results showed a significant improvement for both groups; MAR (Z = − 3.436, p = 0.001) and paper-based (Z = − 3.422, p = 0.001). Also, suggesting that the MAR and paper-based assistance tools helped the participants to recall the names of the PlayStation 3′s console parts. This was proven by the higher mean result in the post-test phase, as shown in Fig. 7, indicating that the participants answered the questions correctly and remembered the names of the console’s parts.

5.2 Performance test

The performance of the MAR and paper-based assistant tools was recorded for both the pre-test and post-test phases. During the pre-test phase, the participants performed the task with the assigned assistance tool according to the group. The duration (time) to complete the task was recorded in the pre-test phase. For the post-test phase, the time and incorrect movement (number of errors) of the participants was recorded; for example, an error is recorded when the participants attempt to remove a part without unfastening the screw first, or attempting to remove a part in the wrong sequence, etc. In the post-test phase, the participants performed the task without the assistance tool, and they were relying on their experience or memory gained from the pre-test to complete the said task.

5.2.1 Task completion time in the pre-test phase between the groups

The participants in the pre-test performed the task consecutively, three times. The collected data from this task were shown to have the best completion times for the three trials. As shown in Fig. 8, the result for the MAR group was (M = 5:52, SD = 1:57), while the result for the paper-based group was (M = 4:00, SD = 0:35). The Mann–Whitney U test confirmed that the MAR group of participants completed the task slower compared to the paper-based group of participants (U = 22.5, p < 0.001) during the pre-test phase.

Fig. 8
figure 8

Results of task completion time

Full size image

5.2.2 Task completion time and number of errors in the post-test phase between the groups

In the post-test phase, the participants started the task once they had completed the knowledge test. The participants were allowed to perform the task one time only, and without any assistance tools. During the time that participants performed the task, the instructor recorded the completion time and the number of errors incurred. Figure 8 shows the results for the MAR group (M = 3:49, SD = 0:47) and the paper-based group (M = 4:01, SD = 0:19). However, the Mann–Whitney U test failed to prove that the MAR group completed the task faster compared to the paper-based group (U = 70.5, p = 0.08). For the number of errors, the MAR group had fewer errors compared to the paper-based group with (M = 0.53, SD = 0.74) and (M = 2.20, SD = 1.32) for the paper-based group as illustrated in Fig. 9. The Mann–Whitney U test provided evidence that the MAR group performed better, having fewer errors compared to the paper-based group (U = 30.5, p < 0.001).

Fig. 9
figure 9

Box plot for the results of the number of errors

Full size image

5.2.3 Comparison of task completion time between pre-test and post-test within the group

The Wilcoxon test for the MAR group showed significance in the completion time to complete the task between the pre-test and post-test phases due to the low p value (Z = -3.067, p < 0.001). MAR group performed faster during post-test phase. The result for the paper-based group was (Z = -0.142, p = 0.89), indicating that there was no significance in the completion time between the pre-test and post-test phases. Figure 8 displays the mean of the task completion time for the MAR and paper-based groups in the pre-test and post-test phases.

5.3 Questionnaire results

The questionnaire developed for the experiment consisted of five parts. Three parts addressed usability (usefulness, ease of use and satisfaction factors) along with other questions for the comfortability, physical and mental stressfulness and user preference on the assistance tool. The usability part consisted of 14 Likert scale rating questions used to prompt the participants to indicate their opinions about the assistance tool they used in the pre-test phase. Each of the factors consisted of at least four items/questions, and the Likert scale ranged between one (strongly disagree) and five (strongly agree).

A Mann–Whitney statistic test was conducted to analyse the mean and standard deviation (SD) for the three factors in the questionnaire: usefulness, ease of use and satisfaction along with other questions, as shown in Table 4. The table shows that the MAR group who agreed regarding the usability of the MAR method was higher compared to the paper-based group (U = 6.0, p < 0.001). The usefulness factor for the MAR group with the mean and SD was (4.47 ± 0.55), whereas, for the paper-based group, it was (2.89 ± 0.64). For the ease of use factor, the mean and SD for the MAR group were (4.18 ± 0.70), and for the paper-based group, it was (3.38 ± 1.18). The Mann–Whitney test indicated the significant difference (U = 31.0, p < 0.001), upon which the participants agreed that the ease of use of the MAR system was better compared to the participants who agreed regarding the ease of use of the paper-based method. For the last factor, the satisfaction factor, the mean and SD for both groups were (4.07 ± 0.73) for the MAR group and (3.15 ± 0.82) for the paper-based group. The Mann–Whitney test confirmed that the majority of participants from the MAR group were more satisfied with the assigned method compared to the participants from the paper-based group (U = 6.0, p < 0.001).

Table 4 Questionnaire descriptive statistics
Full size table

For the assistance tool preferences, a total of 18 participants out of 30 from both groups preferred using the MAR method compared to the paper-based method given its features. The preferences were quite close, which could be due to the inheritance problem of the MAR setup, which made users feel less comfortable using it for a lengthy period.

6 Discussion

Both the data and results obtained from the pre-test and post-test phases showed that the participants in the MAR group displayed better improvement in the knowledge compared to the paper-based group, given the higher number of correct answers. As such, this indicates that the use of the MAR system provided a distinct advantage over the conventional method in gaining or retaining newly acquired knowledge (Fig. 7). Regarding the number of errors, based on the total number of errors for both groups it showed that the MAR group scored fewer errors compared to the paper-based group (Fig. 9). It might be credited to the stereoscopic vision, which provided better depth judgment and the spatial relationship of the different components in the PlayStation 3 console to the user (Holliman 2005). The possibility of encoding more information via the third dimension helped the user in making appropriate decisions (Broy et al. 2012). However, future experiments should consider including an additional condition, MAR without stereoscopic vision, to confirm the assumption of previous studies that indicated stereoscopic vision did not perform well in the virtual reality case (Tawadrous et al. 2017). The visualisation and the animation of the disassembly step were also more evident and clearer compared to the paper-based manual. Therefore, it helped cognitive facilitation, as mentioned by Hou et al. (2013). Accordingly, such systems provide better information registration by transferring the newly acquired information from the sensory memory to short-term memory (STM) as temporary storage, before transferring it to long-term memory as permanent storage. Again, this indicated the advantage in that 3D visual/stereoscopic vision could enhance the user’s understanding (Hung et al. 2017; Majid and Majid 2018).

Furthermore, the developed MAR system was contained 3D objects and animations which helped to motivate the users and encouraging them to pay closer attention to the training process (Solak and Cakir 2016), thereby performing fewer errors in carrying out their set tasks. In other words, the users could remember the disassembly process, acquiring better knowledge retention using the stereoscopic-based MAR system. Based on the observation and feedback, the participants in the MAR group were more enthusiastic and motivated while carrying out the task through experiencing MAR, while the other group performed the task normally, like a routine assembly task. After the MAR group finished the task, some of the participants were curious and amazed at how MAR was employed; even the paper-based group was amazed when they experienced the MAR system. Unlike the paper-based manual, they considered as an old traditional way. Aside from that, the MAR group performed equally fast with the paper-based group during the post-test phase even though it was shown to be slower in the pre-test phase. The slower completion time in the pre-test for the MAR group was due to the participants experiencing and learning about a new technology which they had not previously experienced. Yet, they performed the task equally fast when the assistance tool absented.

Notwithstanding, the questionnaire also measured several factors as mentioned earlier, which were usefulness, ease of use, satisfaction and preferences. The results obtained for the first three factors from the MAR group dominated the paper-based group, as the result was better for both the mean (4.47 ± 0.51 for usefulness, 4.18 ± 0.70 for ease of use and 4.07 ± 0.73 for satisfaction) and p value (0.00). These results also supported the reason why the participants in the MAR group performed better compared to the paper-based group. Overall, the results showed that the stereoscopic-based MAR group demonstrated greater influence and advantage regarding knowledge retention. The participants in the MAR group gained and recalled knowledge better compared to the paper-based group.

However, overall both methods did not provide an excellent comfortable level since all the mean values for the questions were less than 3.7, except when the participants agreed it was not stressful for the paper-based manual (M = 4.00). Some of the participants in the AR groups felt mentally uncomfortable (Q13) as they experienced little drowsiness which is a common virtual reality (VR) sickness following a certain period of use (Kim et al. 2018). Yet their symptoms were not strong, and they were able to complete the experiment. This could be due to the interaction of the person’s head movement which makes the user turn their head to select the button to visualise the assembly step. A different approach such as speech or gesture interaction should be considered, which could help reduce the likelihood of motion sickness. For the paper-based group, a low mean value for physical stress (Q14) was because they did not like to keep flipping through the paper-based manual to refer to the disassembly steps during the process.

In our experiment setup, the result showed that the stereoscopic-based mobile AR systems had a certain advantage regarding knowledge retention. However, the design of the experiment could be elevated to the next level in order to examine the AR technology from an industry perspective and application. Moreover, the product or machine maintenance tasks in industry performed by a worker could be more complex compared to the tasks given in our experiment. Using MAR technology on a more demanding or complex task requires more cognitive thinking which may yield a more significant result or advantage compared to the conventional approach. Here, the NASA Task Load index can be used to determine the complexity of the task. Aside from that, the background and experience of the worker may also be different, whereas the participants recruited in this study were considered as novice users or with little experience in the assigned task and AR technology.

Even though the design of this study was not to reveal the benefits of AR technology for workers with experience on performing similar or routine tasks, it is still an interesting question to be answered by conducting relevant research in industry or in real-life settings. Also, the comfortability of MAR needs to be reconsidered. Based on the mean score of the questionnaire, it suggests that some participants still encountered motion sickness problems even though the wearing time was around 15 min. Therefore, motion sickness would be a big challenge for the application of stereoscopic-based MAR technology in the training as it is run for a longer period. Therefore, future research should ensure that the developed application could reduce the possibility of motion sickness by increasing the frame rate, which would provide smoother interaction or via different interaction techniques, etc., (Weech et al. 2019). On the other hand, when the training becomes longer or complex, user interaction might get difficult and could result in unpleasant feelings to use the MAR system.

Nonetheless, this study has demonstrated the potential of stereoscopic-based mobile augmented reality technology in which user training using this technology could be applied to the novice worker in retaining their knowledge much better in the initial training sessions. An experienced worker may find it is convenient to gain or refresh their knowledge using a paper-based manual. Furthermore, the paper-based manual consumes less time in conducting training. Therefore, paper-based manual training is considered as a good solution for workers who will be beginning fieldwork right after the training. Hence, in such cases, there will be fewer concerns on knowledge retention for a long time. During the exploration in the field task, the worker would be able to retain the skills gained from the training sessions. However, an AR system could be a more practical and cost-effective solution to train the novice workers during orientation training, whereby the training session might last for a week before the new employees are directed to fieldwork.

7 Conclusion

A study was conducted to evaluate the proposed stereoscopic-based MAR system in terms of its usability and performance. The study compared the developed stereoscopic-based MAR application with a conventional paper-based manual in performing a disassembly task for a PlayStation 3 console device. The subjective feedback using a questionnaire was used to collect information based on four main factors: usefulness, ease of use, satisfaction and preferences. The results showed an improvement in knowledge retention for the participants using the developed MAR system compared to the participants using the conventional method. The results support the use of the MAR system, offering greater advantage over the conventional method in acquiring and retaining new information.

In the future, a more comprehensive case study should be conducted. A real-life assembly task involving more complex steps and different background of participants could produce differing results. Although the MAR system will need to be redesigned if the number of assembly steps increases, the prevailing MAR framework is not suitable for this undertaking. Other than that, the MAR system could also be extended by integrating voice commands or gestures as an interaction technique instead of head movement in acquiring friendly interaction for the user to navigate the assembly steps. It can help to reduce the chance of motion sickness symptom. Aside from that, a marker-less tracking algorithm could be adapted. Accordingly, it would be more useful to detect and track the product’s part in offering an intelligent MAR system by informing users on the correct step or error incurred.