The SmARtphone Controller

3.1 Interaction Concepts in AR and VR

The development of new AR and VR HMDs has ramped up over the last years. Major technology and entertainment companies have released the second to third iteration of HMDs to the mass market. Comfort, weight, FoV, visual, and audio quality have been continuously improved. Nevertheless, interaction concepts have not changed significantly. In the domain of mixed reality, interaction is mainly controller-driven, allowing to intuitively grab, throw, or precisely modify the virtual environment. Typically controllers require to be calibrated and are therefore unsuitable for mobile setups. New approaches using ultrasonic and magnetic sensors fused with gyroscope and accelerometer are promising but are still in their infancy. Unlike, commercial AR solutions offer a more fragmented interaction space. Hand gestures, in combination with a head pose, are the most prominent ones.

Today, we use smartphones as a ubiquitous computing device to interact with our environment [5]. We control our home appliances, buy tickets, navigate, or engage with location-based games. Current smartphones comprise numerous sensors that facilitate a good understanding of the environment. Further, devices are becoming more connected than ever and act as a remote interface for current cameras,^[2] speakers,^[3] or toys.^[4] There is a large body of work in which various input techniques have been proposed to interact with virtual elements displayed on an HMD in a mobile augmented environment. In smartphone-enabled handheld mobile AR experiences, direct selection and manipulation of objects give a natural and convenient user experience. However, maintaining visual tracking while holding the smartphone in one hand and interact via touch-gestures with the other hand is challenging [4], [23].

Selection and manipulation in mobile handheld AR got improved by freezing the input [4] or providing additional devices or modified pens [39]. With the availability of optical see-through smart glasses, ubiquitous interaction techniques were investigated. Wearable input is often facilitated through touch surfaces encircling users’ clothes [11], [36] or even fully garment-integrated sensors [25] that are extended by AR. A flexible wearable alternative are sensor-enabled smartwatches. They can provide natural input for short interaction with virtual objects [22], [24]. Further interaction techniques include the use of the user’s body. From mid-air hand gestures to foot-tapping [31], viable solutions for interacting with virtual elements exist.

A different approach was presented by Normand and McGuffin [32]. Instead of augmenting the environment or the user’s body, they augment a smartphone to display additional content around it. A user study highlights their approach’s feasibility, hence promoting to display context-aware information around the smartphones’ display. With hybrid systems that are including AR, VR, or large displays as primary screen technology and an additional handled secondary display or smartphone as a controller enable seamless advanced interaction in mobile context [7], [18]. With smartphones following a “bring-your-own-device” approach, they have been proposed to support spontaneous interaction with large public displays via the user’s smartphone [33]. Practical examples include gaming, were large displays promote multiplayer gaming in social settings [28]. Similarly, Grubert et al. [17] proposed how smart artifacts and the user’s body can be augmented with virtual objects using smartphones. Previous work dealt with the use of the smartphone as an input proxy for other devices. Al-Sada et al. [1] presented the two interaction concepts Input Borrowing and Interaction-Event Mappings which combine smartphone input with other smart devices or AR applications. They show the feasibility of their approach in a user study. However, their user study was employed in a seated scenario.

Detecting the interaction space and virtual objects of interest for manipulation was investigated by previous research. Prominent interaction metaphors for object selection and manipulation are image plane-based or ray casting techniques. Motions of the controller are translated to a spatial ray or are projected onto a plane to support interaction. The inertial measurement unit (IMU) integrated in tangible interfaces [10] or smartphones [18] senses the orientation that is directly translated into the interaction space [21]. Sophisticated and highly specialized handheld controllers were built. Incorporated with a touchscreen, six-degree-of-freedom (6DOF) tracking, and tactile buttons, interaction with immersive applications for VR and AR are viable [29], [35]. Recently, Mohr et al. [30] developed an application that turns a regular smartphone into a 6DOF pointing and selection device to retrofit AR or VR HMDs. They confirmed the feasibility of re-purposing smartphones as an input controller without any hardware modifications. Within this context, Babic et al. presented Pocket6, a smartphone application that uses the integrated tracking capabilities of a smartphone to enable subtle interaction via gestures [3]. Their work shows the technical feasibility of using explicit and implicit gestures for smartphones that can be used to interact with content on the smartphone or virtual content integrated into the environment. The applicability and usability of smartphone gestures have to be considered as well. For example, Serrano et al. [37] found that phone rotations can lead to unpleasant wrist postures over time. Thereby, the usability of gestures has to be evaluated prior to developing and implementing the novel user interface. In this context, Zhu et al. [42] presented design recommendations for the interaction between AR and smartphones. They present BISHARE, an application that enables interaction in AR using a smartphone controller. However, the design recommendations do not evaluate the efficiency, perceived usability, and task load in a user study. A selection of AR and VR devices supported interaction modalities are listed in Table 1 for reference.

Previous work has extensively researched how AR and VR environments can benefit from smartphones as an additional controller for interaction. However, previous work is either limited to the evaluation of gestures or the development of design recommendations, leaving a gap in the evaluation of the overall efficiency and perceived usability of applications when using smartphones in tandem with virtual environments. We close this gap by presenting a user study investigating the smartphones’ feasibility as a second screen and interaction controller for an AR application. We select suitable interaction concepts, provide details about the implementation, present the design and the results of a user study in the following.

Table 1

Overview of out-of-the-box supported interaction modalities of selected AR and VR systems separated into embodied and peripheral (Controller (C)) interaction. ● fully supported, ◑ partially supported, ◯ not supported.

4.1 Object Manipulation

Object manipulation can be separated into translation, rotation, and scaling along the three different axes (i. e., the x-, y-, and z-axis). For object translation, users can move the focused object horizontally (along the x- and z-axes) by touching the screen and swiping horizontally or vertically. Users can adjust the height (y-axis) of an object by double-tapping with and subsequent swipe. For an intuitive translation, a reference coordinate system is created every time the user begins a translation. The coordinate system is created according to the head position concerning the object. Early user tests showed that the decoupling of head and smartphone rotation and position while translating objects is very intuitive and coherent to the user.

For the rotation around the y-axis, we selected a common approach known from map interactions with smartphones. Rotation is initiated by touching the screen with two fingers and then continuously rotate them around each other. To adjust the scale, we adopted the pinch gesture done with two fingers. The space between the fingers is translated directly to the size of the object that is modified. During object manipulation, no information is visible on the display. The interaction design is represented in Figure 2. For comparison, we depicted the elaborated mid-air gestures supported by commercial AR HMDs in Figure 3.

Figure 2

Interaction concept for object manipulation with a smartphone. From left to right: 1) single tap + swipe: translation along x- and z-axes (left/right, back/forth) 2) double tap + swipe: translation along y-axis (up/down) 3) two finger rotation: rotation 4) pinch: scale.

Figure 3

Default mid-air gesture interaction concept for object manipulation. From left to right: 1) Air tap and hold + move hand along the corresponding axis: translation along with hand movement (left/right, back/forth, up/down) 2) two hand air tap and hold + counter hand movement: rotation 3) two hand air tap and hold + move apart: scale.

4.2 Secondary Screen Support

Supplementary to object modifications, AR applications often require input on free-floating or space anchored 2D GUIs. Our smartphone supported approach leverages two different possibilities to interact with these kinds of interfaces. First, by implementing a simple remote-like controller similar to the object manipulation. Users can swipe and tap to interact with the AR display space presented user interface elements like buttons, slides, or checkboxes. In the second approach, the entire user interface is transferred on demand onto the smartphones’ display and allows seamless and direct touch and swipe interaction with the represented elements.

5.1 Smartphone Application

We developed a native Android application to provide seamless input and output. Touch input, swipes, and gestures are sensed and directly sent to the AR HMD. User interface elements can be displayed at the request of the AR HMD. Any GUI input is forwarded to the HMD accordingly. For bidirectional communication between both devices, a Bluetooth radio frequency communication channel is established. Touches, swipes, and gestures are transferred via this channel to manipulate visible objects in AR. Bidirectional status messages are transferred for UI input, states of the application, or haptic feedback control commands. We used Google’s Protocol Buffer because it provides a fast and platform-neutral serialization protocol of the structured data.

5.2 HMD Application

The HoloLens displays the AR environment and processes any incoming interaction commands from the Smartphone application. For a smooth and enhanced user experience, any continuum user-input is low pass filtered to remove any jitter. If a selected virtual object contains a context menu, the presentation of this user interfaces is triggered on the connected smartphone. Both the HoloLens application and the AR environment are implemented using the Unity game engine 2018.4.7 and the Mixed Reality Toolkit v2.

6.1 Subjects

We recruited 24 participants (12 female, 12 male) via our universities’ mailing lists. Participants received either 10 EUR or course credits as compensation for their participation. Four had previous AR experience, one of them used AR glasses for professional purposes. The study received ethics clearance according to the ethics and privacy regulations of our institution.

6.2 Apparatus

The apparatus for this study comprised a Microsoft HoloLens and the Google Pixel 2 XL running Android 9. The smartphone has a bright, high-resolution display (538 PPI) with a presentation and interaction area of 136×68 mm. Both devices are connected via Bluetooth and run our developed applications presenting the different stimuli and logging the data. The developed smartphone application serves as an input controller as well as a secondary display. For the baseline, only the HoloLens with the build-in mid-air hand gesture support was facilitated. Our experiment was conducted in a room with controlled light conditions for consistent visibility of holograms and a free interaction area of approximately 3×3 m.

Figure 4

The first experiment includes manipulation of the position, rotation, and size of colored cubes. Arrows and lines help to compensate for the limited field of view and enhance orientation in space. The first condition includes manipulation via mid-air gestures (left). The second condition of the first experiment involves multi-touch as an input modality (right). Participants can manipulate colored cubes by swiping and multi-touch gestures on the smartphone.

6.3 Procedure

After welcoming the participants, we asked them to sign the consent form. We explained the course of the study, all devices, and the interaction concepts to the participants. Afterward, we adjusted the AR glasses to the participant’s head and ensured that the participant can comfortably perceive the entire display area. In the last preparation step, we ask the participants to start our application, which guides them through the study. The application starts with a tutorial and aids the participant in getting used to the different interaction concepts. Participants could freely practice the first input modality until they understand them and feel comfortable during the tutorial. Specific questions regarding the input and study were answered, and the main task was explained. Participants were requested to finish the tasks as fast and as accurately as possible. Then participants started with the object manipulation task. After manipulating all 12 objects (three sets of four objects), they had to fill out the NASA-TLX [19] questionnaire on a dedicated laptop. Participants could take a break at this point since all tasks were performed in a standing position to minimize fatigue. The participants continued with the second task using the same input modality. Again, a tutorial was presented introducing the new task and allow the participant to practice and familiarize themselves with a different function. After accomplishing the new tasks, participants filled out the NASA-TLX questionnaire. Finally, they repeated both of the experiments with the other input modality.

The input modality and the tasks were presented in a counterbalanced order using a full Latin square to prevent any sequence or learning effects. Throughout the study, we logged all interactions with the system for subsequent offline analyses. After successfully finishing both experiments, we asked for comments about the user experience and which input modality they ultimately prefer. Participants completed the study, including the debriefing, within 40 to 55 minutes.

6.4 Experiment 1: Object Manipulation

For the first experiment, we designed an object manipulation task in which participants had to select, translate, rotate, and scale different colored cubes. Four virtual cubes with an edge length of 50 cm were placed in front of the participant. After the selection of a cube, a white line guides the participant to the ghost representation of the cube representing the target state. Participants were asked to perform the manipulations using the given input modality as precise and fast as possible. Both conditions are visualized in Figure 4. Task complexity was increased through three sets of four cubes. The first set only includes a translation to achieve the target transformation. The translation requires movements of the cube between 25 cm and 125 cm along all three axes in a predefined direction. The second set of four cubes additionally includes rotation. The rotation offset was between +/−30 degree and +/−150 degree, whereat participants could rotate the cube either way in one seamless interaction. Lastly, participants had to translate, rotate, and scale each of the cubes, respectively. Matching the size of the target cube required the participants to set the scaling factor between 0.6 and 1.8, creating cubes between 30 cm, and 90 cm edge length.

We measured the accumulated task completion time (TCT) starting from the first modification till the last modification of each cube since we were only interested in the object manipulation performance. Further, we recorded the accuracy by calculating the euclidean distance in cm, absolute rotation in degree, and scale offset in percent between the ghost representation of the target cube and the cube placed by the participant. Finally, we assessed the task load through the raw NASA-Task Load Index (raw TLX). In the first experiment, we recorded a total of 578 object manipulations.

6.5 Results 1: Object Manipulation

Figure 5

Mean values for TCT, relative error, and raw TLX Score for each condition of the first experiment. Error bars show standard error of the mean (SE). Asterisk indicate statistically significant differences between conditions.

We analyzed the TCT and accuracy concerning the position, rotation, and scale deviation. We further assessed the perceived task load using the NASA-TLX questionnaire. We used a repeated-measures t-test for statistical comparison. The significance level for all comparisons was set to α=.05. The results of the first experiment are graphically depicted in Figure 5.

6.5.1 Task Completion Time

The aggregated TCT for all manipulations of mid-air gesture input (M=453.47, SE=34.84) was significantly higher than multi-touch input (M=393.53, SE=25.01), t(23)=2.341, p=.028, r=0.438. The effect size estimate indicates that the difference in TCT created by the input modality was a medium effect. To understand the strength of the smartphone input we subsequently analysed each of the three task complexities individually using t-tests. We found significant differences between mid-air gesture input (M=173.47, SE=15.74) and smartphone input (M=135.42, SE=10.52), t(23)=2.799, p=.010, r=0.571, for the medium complex task including translation and rotation but no significant differences for the others tasks (all p>.05).

Figure 6

The conditions of the second experiment. Participants are either controlling the GUI via mid-air gestures (left), touch gestures on the smartphone or the GUI is displayed on the smartphone itself (center/right) while the task and elements are displayed in AR.

6.6 Experiment 2: GUI Interaction

For the second experiment, we designed a menu with a list of modification options for a virtual 3D object. The participant had to interact with the menu and change the settings according to the presented information. Therefore, the target state was textually shown next to the object to modify. The menu comprises drop-down items, radio-buttons, slider, and regular buttons.

In this experiment, input has three levels. Additional to mid-air gesture and multi-touch, we introduce multi-touch display. In that condition, the menu was directly presented on the smartphones’ display, and the participant could directly interact via touch. In the other conditions, the menu was anchored in space, facing the user, and had to be operated via mid-air or multi-touch gestures. Both visualizations are represented in Figure 6. In total, participants had to complete 36 tasks, 12 in each condition.

Again, we measured the TCT and the raw TLX score. Since the correct input for all parameters was required to complete the task, we did not measure an error rate. TCT was measured from first to last input for each sub-task individually. In total, we recorded 864 menu interaction.

Figure 7

Mean values for the TCT (left) and raw TLX Score (right) for all three conditions of the second experiment. Error bars show standard error of the mean (SE). Asterisk indicate statistically significant differences between conditions.

6.7 Results 2: GUI Interaction

We statistically compared the TCT, and the raw TLX, using one-way repeated measures analysis of variance (ANOVA). The results of the second experiment are displayed in Figure 7.

7.1 Limitations and Future Work

Our smartphone controller approach currently supports only a subset of interaction paradigms necessary to fully interact in generic AR applications. However, we demonstrated the feasibility and potential of multi-touch and a secondary touch display in AR environments. Extending our approach with already existing positional tracking solutions [8], [30] would enable fluent, precise, and convenient input in mobile or spontaneous interaction scenarios. Additional sensors like accelerometer, gyroscope, or proximity could be incorporated to further increase the interaction space.

Our approach uses a well-developed high-precision touch interface for interaction. In contrast, mobile mid-air gesture tracking is more complex and less accurate through different factors that are given by the spatial tracking and temporal camera resolution. Our participants are used to smartphone interaction and gestures on multi-touch displays. Mid-air gestures are less common in everyday interaction with computing systems. Hence, our results may not be generalizable for users with extensive hand gesture experience that may be less prone to fatigue. In our study scenario, the relationship between the secondary handheld display and augmented space is evident. In more complex scenarios, methods need to be implemented to keep it comprehensible to the user when and how information is displayed on the secondary screen. For future research, we envision an investigation on how our smartphone-based controller performs outside the lab and whether other modalities outperform our interaction concept, e. g., during micro-interactions on the move.