Keywords

1 Introduction

Quantitative and qualitative data collection utilizing questionnaires (whether standard or specialized) during user studies for in virtual environments are conducted post-task via electronic or paper-based mediums. While this produces viable data, it lacks specificity of time and actions, rather serves more as a post-reflection. This type of data collection may miss subtitle responses on more specific actions or reactions that are less obvious to or missed by a user in real-time during their task. Particularly in a virtual environment, this becomes more difficult due to the immersive display, closing off the user from the real world. In this paper, we present the use of a custom input device controller for dynamic real-time data collection of questionnaires or other subjective inquiry for use during tasks in virtual reality using head-mounted displays. While automatic logging of user data in a virtual environment is standard practice, there is little work done to provide users with real-time input responses outside of an experimental task [1, 3, 5]. Other experiments have utilized a button on an input controller to identify a response [2], but these actions have to be done as a separate component (not in real-time during the task) and may take too long to control visual widgets. We conducted a user study utilizing our technique and present the results of the feasibility evaluation of the data collection activity. We used two types of subjective questionnaires, a specialized questionnaire to gather a wide-range of valued responses and a 7-point standard presence questionnaire- the Witmer-Singer presence questionnaire [4]. Our findings present that is not only feasible to utilize such a device for data collection during a task in virtual reality, but that the data collected demonstrates a much more rich and detailed perspective on the feedback from users in real-time. Our results have the potential to improve subjective user data collection for user studies conducted using immersive virtual reality. This paper is organized as follows. Section 2 describes the related work and background of our work. Section 3 describes the description of our custom device controller i.e., KnobCollector. Section 4 presents our experimental study and Sect. 5 describes results of our study and Sect. 6 presents our conclusion and future work.

2 Background and Related Work

Technological advancement has introduced different methods for how quantitative and qualitative data of an experiment is collected and analyzed. In general questionnaires, observations, and interviews using a paper or an electronic medium are typically used to collect subjective data of a user experiment. This method of collecting subjective data usually fine, but in the context of virtual reality or when using immersive systems, a user must exit a virtual environment or take off an immersive system to provide the answers which might affect the response provided [6]. Researchers have designed tools and techniques for data collection while a user is within a virtual environment to maintain the level of immersion.

2.1 Button and Controllers

Researchers [7] investigated the effects of feedback delay on the qualitative subjective experience of virtual reality. They used the thumbstick on the left Oculus Touch controller [8] for scaling task as an input for data collection for scaling task. This method supported a user to provide subjective feedback of the experiment while the user is experiencing the virtual environment. This method requires the participant to train to which button in the controllers to be used, however our technique provides a dial controller where the data input is mapped more intuitively to rotate each direction to increase or decrease variable data on a scale. Furthermore, researchers [2] presented the use of a button on an input controller to provide the input for data collection. However, this use of the button to provide must be done separately and not in the real-time during the task which might affect the response of the participant.

2.2 Smartphone

Tsaramirsis et al. [9] presented navigating in virtual environments using smartphone sensors. The data collected from the smartphone sensors along with the machine learning technique is used to help the user to navigate in the virtual environments. This method only takes sensor as the data but cannot be used for the collection of the data for subjective response in real-time. Furthermore, Laaki et al. [10] showed a situation where data collected from the smartphone device is used to augment the virtual world along with the real-life data. This method of collection is different from our research work as our KnobCollector is used to provide data to the subjective questionnaire in the real-time.

2.3 Virtual Avatar

Hasler et al. [11] used a human-controlled avatar to interview the participants within the virtual world, Second Life [12], about their religion. This use of an avatar for interviewing the participants allowed the user to be within the virtual world and provide the answers to the questions. Furthermore, this provided a way for face-to-face communication, but our work is more directed towards providing input for data collection individually using a tool and is flexible move and to integrate with different immersive virtual environment.

2.4 Virtual Interface

Bell et al. [13] designed and developed a virtual data collection interface (VDCI) which used a virtual assisted self-interview (VASI) method in Second Life [12] to collect subjective data while a user is in the virtual world. This tool allowed to survey within the virtual world for collecting data while being immersed in it. Faleiros et al. [14] designed virtual questionnaire for a virtual platform to collect data of Germans diagnosed with Spina Bifida (SB). The virtual questionnaire consists of 57 questions, and it was a Likert Scale question. While these will enable real-time data collection in a virtual world, typically online virtual environments are not experienced using an immersive system so that a user can utilize standard input devices and see the virtual world as well as the devices. Our technique helps a user collect data while fully immersed in a virtual environment and cannot see the device in the real world.

3 Custom Device Controller – KnobCollector

We present our novel device, KnobCollector (see Fig. 1), that can be used to collect subjective quantitative data in real-time while a user is experiencing a virtual environment using an immersive virtual system. To prototype the controller (see Fig. 1), we implemented an Arduino UNO board [15], used a dial for collecting variable data, and a laser-cut 3D printed a case for more ergonomic angle placement of the device. This device can be used on the table (as used in this experiment) but also attached to the body. In either case, it allows for comfortable and easy control of the device while using a head-mounted display. This device controller is lightweight and can also be held in a user’s hand while interacting in CAVE [16], IQStation [17], HMD, or another different immersive virtual environment while experimenting. It was important that the user could ‘feel’ the knob and comfortably turn the knob left and right so as not to take away from the virtual reality experience. As such the knob has ridges to grasp and feel the knob better. This device currently uses a USB cable to connect and transfer the data to the computer, however, in future we plan to add a feature for wireless connection with the computer. The custom device controller knob was calibrated to provide a custom range of input (calibrated by a user). We customized each rang for the standardized questionnaires that we used in this experiment: from 1 to 5 for a customized System Usability Scale (SUS) [18] questionnaire and from 1 to 7 for the standard Witmer-Singer Presence questionnaire [4].

Fig. 1.
figure 1

KnobCollector, a custom input device for real-time subjective data collection from a user.

Full size image

4 Experimental Study

Each participant was presented with the experimental study environment and was asked to complete the task in a virtual environment. The participant used our KnobCollector device after each trail while they were in the virtual environment to provide their responses as input to the SUS and Witmer-Singer questionnaire in real-time during the experience.

4.1 Experimental Environment and Apparatus

We used the Unity3D game [19] development platform (version 5.4.0) and C# programming language to develop the experimental user study. An Arduino UNO board [15] was used to create the custom device controller which connects to the dial for collecting input data. A package, Ardity [22] was used to move the data from the Arduino board to Unity, bidirectionally, over a COM port. A USB cable was used to connect to the computer for transferring the data collected while rotating the dial during the experimental study. The virtual slider that is shown in the experimental environment is developed in the Unity3D which corresponds to the rotation of the dial.

4.2 Participants

We recruited participants by sending out information using emails, flyers and mouth-to-mouth information. All the participants recruited were from of the University of Wyoming. We had a total of 20 participants which included 12 females and 8 males. The participants age ranges from 19–51 years (μ = 25.55, σ = 8.28)). Each participant was screened for proper 3D-vision using Butterfly Stereoscopic Test [20]. To compensate for the time, each participant was paid $5 and was entered for raffle of a $50 gift card.

4.3 Procedures

This experimental study was approved by the University of Wyoming Institutional Review Board. Each participant signed the consent form, and they were presented with the Butterfly Stereoscopic Test for proper 3D-vision. After the completion of 3D-vision test, participants completed an online pre-questionnaire (in the real-world). During the experiment, participants completed the experimental task by which they each experienced a virtual environment and then provided input of their responses to each prompt during the experience using our KnobController device. After the experimental study was completed, each participant completed a post-questionnaire. The total time for the experimental study was approximately 1 h.

For the experimental task, participants were seated at a table with our KnobCollector device in front of them. Participants were exposed to a virtual reality environment and asked to do complete task to explore objects in the virtual reality environment. Participants were presented with the virtual environment using an HMD (HTC Vive [21]). To avoid distraction for the participants, we masked the sound of our moving apparatus by providing headphones and playing background elevator music. After each trial, the data was collected in the experimental environment with the use of the KnobCollector (combined with virtual sliders for visual feedback). The ratings were input to the experimental study environment by using the custom device controller - KnobCollector (see Fig. 1). The participants turned the dial left to select a lower value and turned it right for a higher value. As the participants turned the dial, the experimental user environment provided feedback about their current selection by showing a moving slider (see Fig. 2). Figure 3 shows a mock participant using the described setup.

Fig. 2.
figure 2

Virtual slider movement corresponds to a user’s input change of the device controller.

Full size image
Fig. 3.
figure 3

A mock participant interacting with the experimental user study environment. The participant wears an HTC Vive to view the virtual environment and her left hand is rotating the dial to provide the input for the data collection during the experiment.

Full size image

5 Results and Discussion

Each participant provided their responses as input using our KnobCollector device. The participant positioned the visual slider by using a dial provided in the KnobCollector. During the experimental task the values collected could vary between 0 and 100 but participants could not see the values only the point at which the slider was on the virtual widget (Fig. 2) in the HMD. The values were then converted based on the calibrated scale for each type of questionnaire used or rating for data input. Ratings were collected during their experience and after. Without use of the KnobCollector we might only obtain a final rating for each participant, or at the very least a final rating after each trial, resulting in a summary of ratings across the trials as in Fig. 4. With data collection using KnobCollector we can gain so much more. Feasibility is demonstrated for collecting a rich amount of data over time that can show critical incidents as in Fig. 5. Additionally, as seen in Fig. 6, there is a wide variability and richness of data collected over time in the ratings for each trial.

Fig. 4.
figure 4

User ratings after each trial, collected after a user’s experience.

Full size image
Fig. 5.
figure 5

User ratings over time showing critical incidents using KnobCollector during an Immersive Virtual Environment experience.

Full size image
Fig. 6.
figure 6

User ratings over time colored by trials using KnobCollector during an Immersive Virtual Environment experience.

Full size image

5.1 Witmer-Singer Presence Questionnaire

Using the standardized Witmer-Singer Presence Questionnaire, we collected the impressions of the participants while interacting and conducting the task of the experimental user study in the virtual environment. The results are listed in Table 1. These results suggest our KnobCollector using the dial did not detract from the experience of the participants, as it did not distract them from the tasks they performed; and even, from their post-study comments, we found they enjoyed using the dial for providing the input for the data collection. Questions prefixed with an asterisk are expected to have a mean close to 3 as they do not have clear applicability to our study. The highest ranked questions of the questionnaire are listed in Table 2. Conversely, the lowest ranked questions are listed in Table 3. Furthermore, below we present additional questionnaire used for the Witmer-Singer Presence questionnaire.

Table 1. Results of the Witmer-Singer Presence Questionnaire, ratings from 1 to 7.
Full size table
Table 2. Highest ranked questions of the Witmer-Singer Presence Questionnaire.
Full size table
Table 3. Lowest ranked questions of the Witmer-Singer Presence Questionnaire.
Full size table

In our comparison, we found similar results on the sense of presence and user perception of the objects in virtual reality. Through this data, we can validate the feasibility of the technique can be used instead of or in combination with the post-task data collection. Furthermore, our paper will present the detailed results that with more rich data collection and in real-time while a user is fully immersed in a virtual environment, we were about to derive more detailed differences in presences and user response based on time. Small periods during the experience revealed interesting results in relation to user response time, user interpretation of the virtual reality environment, and user initial reactions versus reactions based on exposure over time. Our results may have implications on how designers and developers design virtual reality environments as well as a practical and useful tool for researchers conducting user experiments in virtual reality who desire more detailed data collection and critical events based on time. Furthermore, we believe KnobCollector can be used as an tool for data collection for experiences in immersive virtual environments. One idea is that the level of presence will change while a user is experiencing the virtual environment as opposed to completing the questionnaire after the experiment as he/she has completed their experiences of the virtual environment. KnobCollector will provide an effective medium to complete the pre/post-experiment data collection without having an effect of any external real-world factors (Table 4).

Table 4. Additional Witmer-Singer Presence Questionnaire used for experimental user study.
Full size table

6 Conclusion and Future Work

We presented our custom device controller, i.e., KnobCollector, for dynamic real-time collection of subjective user data during experiences or experimental studies in real-time while in an immersive virtual environment. The KnobCollector provides rich data during an immersive virtual reality experience that can help identify critical incidents as in Fig. 5 and further breakdown of data by trials or other conditions as in Fig. 6. KnobCollector provides a practical tool for researcher and scientist while conducting a user experiment in the virtual reality to collect subjective data who desire to have their data collected while the participant is within the virtual environment. The tool has been easily mapped physically to the directionality of standardized questionnaires that can be presented virtually in the virtual environment. The results presented are promising in terms of providing confirmation of feasibility that this device can be used to provide a richer set of data. The results are exciting as the participants found the dial in KnobCollector used for collecting the input data during the experiment did not distract them from the immersive experience while doing the task. Furthermore, from the post-study feedback, we found that participants enjoyed using the KnobCollector to provide their subjective rating responses for each questionnaire during the task.

In future work, we also plan to conduct a user study to compare KnobCollector with different other methods of subjective data collection. Furthermore, we will work on developing the KnobCollector so, that it supports wireless connectivity which will help provide a user with flexibility to move around in a larger immersive virtual environment while interacting with the task.