Keywords

1 Introduction

Augmented reality (AR) systems became popular in recent years, with application in several areas such as education [1, 2], health [3], industry [4] and entertainment [5]. The main idea behind AR is to combine virtual and real information in real-time, registering 3D information [6]. This idea seems to be simple in theory, however it involves several aspects such as rendering the virtual objects and capturing real environment data in order to combine both pieces of information. Azuma [6] defines three characteristics of AR, such as:

  • three-dimensional (3D) space composed of real and virtual elements, with predominance of the real over the virtual;

  • real-time human-computer interaction;

  • 3D registering, with alignment among real and virtual elements.

Due to problems in rendering and capturing, commonly caused by technological limitations, such as resolution, processing power and noise of input devices; and aspects from real environment, such as lighting and real objects and camera motion, different errors may be generated by the system during the human-computer interaction. These system errors can differently affect the user perception of whether the virtual information is real or not.

Several works examined how rendering errors affect the user perception [7–10]. Sanches et al. [11] studied the system error perception in video based avatar rendering issues. Also, [12–14] studied perception in Virtual Reality systems with haptic interaction.

However, little has been investigated about how the 3D registering errors affect the user perception of virtual objects placed over real surfaces. We addressed this problem, studying different types of common errors inside RGB-D registering based on three-point RANSAC pose estimation method [15], which enables fast registration of rigid object and is widely used.

Our main objective is to analyze system errors of 3D registering according to users perception, through subjective tests. Our contribution is the analysis of how system errors are perceived by users and which errors more or less affect the perception. Therefore, analyzing how high or low the different AR system errors should be in order to achieve a better user perception, even with these errors, which would help in the development of these interactive systems.

2 Related Work

User perception of virtual information realism is a central focus in several works. Gkioulekas et al. [7] presented a user perception study on the role of the phase function in translucent objects. Also, Jarabo et al. [8] and Křivánek et al. [10] studied realism in global illumination rendering, Jarabo et al. [8] investigated the perception of complex dynamic scenes, such as crowd movements, and Křivánek et al. [10], the effect of global illumination approximation impact on perception.

MacDonnell et al. [9], Gelasca and Ebrahimi [16] and Sanches et al. [11] studied user perception related with human avatars, synthesized and captured. MacDonnell et al. [9] studied the effects of human avatars and realism perception, and pointed out the uncanny valley effect [17] in avatar rendering. Gelasca and Ebrahimi [16] and Sanches et al. [11], in turn, studied the artifacts of background segmentation of user videos. Where, Sanches et al. [11] focused on AR and its impacts on user perception.

Several works in haptic interaction research also use objective evaluations with psychophysical models to determine user perception when a user is submitted to tactile stimuli [12–14], analyzing the tolerance in the occurrence of certain noise or errors in haptic signals.

As spotted by the works presented above, several aspects related with perception in AR [6] were studied isolatedly, such as rendering or interactions. However, little has been studied about another central aspect of AR, the registration of real and virtual objects. Sanches et al. [11] directly studied perception in AR systems, yet, their work relied on rendering aspects. Our work addresses this gap, investigating perception in AR registering focusing on common system errors that can happen, analyzing how these errors affect the perception in AR.

3 Registering Method and User Test Stimuli

Registration of view in AR systems can be achieved by several approaches [18–20], and recently RGB-D cameras have become widely available. These cameras enable faster and easier methods of pose estimation. One pose estimation method based on these cameras are the three-points based RANSAC based methods [21], used in several works, such as Henry et al. [15]. These methods pose estimate known scene or objects based on correlation of known feature points with observed feature points, extracted using feature point extraction methods such as SIFT [19] or SURF [20]. In this work, we will use this three-points based RANSAC pose estimation with SURF feature points in order to analyze the method performance through system error based on user perception.

Two common system errors in this method are the noise present in RGB-D depth map, caused by the capturing system, and 2D (two-dimensional) position error of detected feature points in the image, caused by mismatches or position error of the feature matching algorithm. Also, clustering of the detected feature points, generated by occlusions of the object or because of the original object texture, is another error. This clustering (Fig. 1(a)) can cause pose estimation errors, since the points are not spread all over the object surface, small errors in captured data can thus affect the pose error more.

Fig. 1.
figure 1

Feature point clustering.

Full size image

In order to analyze the user perception, two controlled experiments were conducted. Here, 150 different ground truth images were pre-rendered with different plane image poses, in order to decrease pose related perception bias. For each error value, all the 150 images were re-rendered with the result returned by our pose estimation algorithm with the error injected, as in Lepetit et al. [22]. In the user test, some of the 150 images are randomly shown with their ground truth images for each error value.

Experiment 1 was executed to analyze the effect of two types of errors themselves over the registration, without error combination. Two tests were preformed, varying the depth (DEPTH) and 2D position (POS_2D) noises with 7 different values for each, as presented in Fig. 2. The depth map errors are injected as a Gaussian random noise of mean \(\mu = 0\) and standard variation with 7 values varying from \(\sigma = [0, 20, 40, 60, 80, 100, 120] mm\). In order to add the 2D position noise, we added a two-dimensional Gaussian random noise of mean \(\mu = 0\) and standard variation with 7 values varying from \(\sigma = [0, 16.6, 33.3, 50, 66.6, 83.3, 100] pix\), similarly to depth noise. This experiment indicated how these two errors are perceived by users, and how they differ from computed absolute errors.

Fig. 2.
figure 2

Experiment 1 - images and error examples. First row: depth map noise; second row: pose estimation result using the noise added depth map; third row: 2D position noise; fourth row: pose estimation result. First column: the ground truth values and poses. Second to seventh column: the increase in noise values and their resulting poses, 20 to 120 mm of noise for depth map and 16.67 to 100 pixels for 2D position noise.

Full size image

In Experiment 2, 4 variables were chosen: depth (DEPTH), 2D position (POS_2D), clustering type (CLUSTER) and object rendered over the plane (OBJ). We varied the object, since one hypothesis was that the object can affect error perception, providing guides and clues t to the error. Figure 3 shows some examples of pose estimation results with these combined errors.

In this test, we varied the depth map noise variance with three different values \(\sigma = [0, 20, 60]mm\). Also, the 2D position noise was modified with three different variances of \(\sigma = [0, 16.6, 50] pix\). For the clustering, we used two different types, the spread and in line clustering as shown in Fig. 1. And, lastly, we chose two different objects to be rendered, the Happy Buddha object from the “Stanford 3D Scanning Repository”Footnote 1, as an object that is taller but does not occupy all the space of the pose estimated plane object; and a red grid that perfectly fits around the plane object, as shown in Fig. 6.

Fig. 3.
figure 3

Experiment 2 - images example. First row: Buddha as object, second row: red grid as object. First column: ground truth pose; second column: depth map noise only; third column: 2D position noise only; fourth column: combined depth and position noise; fifth column: noise combined with line clustering (Fig. 1(a)).

Full size image

In Table 1, we show the user test stimuli used in the two user tests. It is worth pointing out that our objective here is to evaluate system errors and characteristics, not effects caused by environment setting. Therefore, screen, lighting conditions and other physical environment are not chosen as stimuli.

Table 1. Full stimuli list used in the two experiments.
Full size table

4 User Test Setup

The experiments were conducted in controlled physical environments, using common video monitors and an automated system to collect users opinions. A simple yes-no psychophysical task was used to compare two images. These two images were presented side by side on one screen, being an image to represent the ground truth and other with noise or equal to ground truth. The comparison of the ground truth with itself allows to avoid response bias, ensuring a balance between the two answers. The pairs of images were shown in random order (70 pairs in the first experiment and 114 in the second experiment).

To each user was asked to select the option “Equal” or the option “Different”, clicking on one of the buttons located at the bottom of the screen after observing the two images. A time for response was not specified; however, users should answer quickly. Figure 4 presents an example of the virtual environment used in the tests with the screen and the two images, as well as the buttons to register the user opinion. In each experiment, 17 subjects participated, the major part composed of students and professors.

Fig. 4.
figure 4

Test environment example.

Full size image

In order to test statistically significant differences in answers to different scenarios, we chose one-way ANOVA (Analysis of Variance) for first experiment, aiming to analyze the effects of two variables separately (DEPTH and POS_2D); and four-way ANOVA for the second experiment, which allows analyzing the effect of one of the four variables (DEPTH, POS_2D, CLUSTER and OBJ), independently of the other variables (called Main Effects); as well as the effect of one variable when there is dependence on the level or levels of the other variables (one or more variables, with analysis of two, three and four variables), called Interaction Effects. After finding the effects (main and interaction), Tukey post-hoc test was selected to compare pairs of means and to determine the causes of these effects in both experiments.

5 Results

Figure 5 shows the results from the first user experiment. Here, the mean values of the percentage of times in which users perceived the errors inserted in the images are shown, as well as the ground truth and the computed pose errors. These data were processed with one-way ANOVA and Tukey post-hoc test, where null hypotheses were false in both cases, depth (\(F = 30.97\), \(p = 0.0224e^{-30}\)) and 2D (\(F = 26.75\), \(p = 0.0311e^{-26}\)).

Fig. 5.
figure 5

User perception for Experiment 1 and computed errors.

Full size image

From these graphs, it is possible to observe how the decrease of registration perception is different from the absolute computed error increase. The depth errors values are perceived almost as an inverse exponential function, from graph (Fig. 5(a)), where the computational error increases almost linearly. We could deduct that depth map noises are easily perceived by the user in this setup, once it creates rotation errors at the object pose estimation.

Two-dimensional position errors are also perceived differently from the computed errors (Fig. 5(c)). However, differently from depth map errors, the position noise decreases more slowly and the computed absolute error increases almost exponentially. This could be due to the nature of the pose error returned by this noise. Since only the position is varied and the depth map is clean in this test, the resulting error appears as a slide of the virtual object over the plane object, as seen in Fig. 5. With this, small errors are weakly perceived by the user.

Results of the second test, in which four different variables were chosen as stimuli, were processed with four-way ANOVA and Tukey post-hoc test. The full table of rejected null hypothesis, with their F and p-values, and returned post-hoc analysis are listed in Table 2, and part of these significant results are illustrated by graphs in Fig. 6.

Fig. 6.
figure 6

User perception for experiment 2.

Full size image
Table 2. Significant results from experiment 2.
Full size table

One of the significant effects found is that the rendered object is one factor that changes user perception, once the null hypothesis of OBJ was rejected (\(F = 211.17\), \(p \approx 0.0\)) (Fig. 6(g)). This result shows that the virtual object to be included in the real world significantly affects registration perception. This can be explained by comparing the images in Fig. 3. Once the grid object is more aligned with the plane object to be pose estimated, the user has more clues to correlate both objects poses, making it easier for the user to find errors which the registration. This effect is clearer specially in POS_2D errors as shown in Fig. 6(f), where the difference in perception across errors decays with grid OBJ.

One change observed from the first experiment was the perceptions related with POS_2D error. In Experiment 1, POS_2D errors are less perceived by users; however, in Experiment 2, POS_2D perception has a strong decrease with error increase (Fig. 5(b)). This can be explained by the point that POS_2D error perception could be strongly increased when combined with other types of errors, as in Figs. 6(f) and 6(h). Figure 6(h) allows observing that DEPTH errors are strongly affected by the POS_2D errors, meaning that combinations of both errors can generate worse perceptions.

Finally, one interesting issue is that CLUSTER related errors did not strongly affect user perception, although the null hypothesis was rejected. In Fig. 6(c), it is possible to observe that the change in perception is not strong with our clustering types, and from Fig. 6(d), the CLUSTER type was significant only for middle values of depth errors. Even the null hypothesis of the interaction effect of POS_2D and CLUSTER was not rejected, showing that clustering does not play a strong role in registration perception, with the level of clustering we applied to this work (illustrated in Fig. 1).

6 Conclusion

We discussed how system errors in the registration method applied to AR systems based on RGB-D cameras can affect users perception. Although, this registration process is one of the central aspects in AR systems, little about its perception has been studied. Based on yes-no psychophysical tests and ANOVA analysis, we spotted several traces of perception in AR, such as the role of relation between rendered object and real object, which directly affect perception.

Also, we address how some system errors appear in the object pose and how this affects the user perception. Effects such as depth map errors alone are more perceived than feature points position errors; however, when combined with other errors, position errors cause a huge decrease in perception. Another issue is that user perception is not identically related with the absolute computed error, having sudden decrease with even small changes in some errors.

Although our tests cover the major system errors in the RGB-D registration method, more tests need to be applied in future works, in order to fully explore registration related perception issues in AR. For example, how and why this real-virtual objects relationship affects perception, or effects of the test environment, such as screen resolution and illumination issues.