Keywords

1 Introduction

Military personnel require training throughout their career. Programs, such as the Warrior Leader Course (WLC), provides Non-Commissioned Officers (NCO) with training for enhancing squad leadership skills. The WLC is a four-week experiential learning course covering topics such as tactical operations, combat orders, casualty evacuations and land navigation through the use of classroom training, field exercises, and subsequent assessments [1]. Advancements in technology have provided a method for supplementing traditional classroom training through virtual platforms. A common part of utilizing virtual platforms is the user’s controller interface for a system: an input device supporting maneuvers within generated scenarios, through some form of manipulation. Virtual platforms such as Virtual Battlespace 3 (VBS3) support scenario-based training that can replicate field exercises using the virtual environment. It is vital that the most appropriate input devices are used to maximize effort on completing the training task, and minimize throughput time. This paper identifies several input devices as possible alternatives to the traditional keyboard and mouse, focusing on human performance within a Virtual World (VW).

The user interface is the first interaction between a product and the Soldier, making the interface a critical training attribute to be examined. If this step of engagement is difficult, problematic, or confusing to its users, the product risks not achieving its intended purpose or goal [2]. Several methods of input (such as a traditional keyboard and mouse or gaming console controllers), or interfaces, can be used for virtual platforms. Research has expanded to testing the usability of wearable controls, such as a head mounted display [3]. Ultimately, a highly intuitive device is sought to reduce the time impediment of learning or performing device operation, therefore allowing more time for conducting training tasks.

Pairing a device suitable to a task allows easier interaction, and video game users have been found to prefer an input device that fits the type of gameplay [4]; although a naturally interactive device might not necessarily bode well for performance [5]. User preference can also depend on familiarity and comfort [4, 6]. Unfortunately, the simulation interface is oftentimes an afterthought during the design and development phase of a system; it is only after initial user testing that the interface is substantially analyzed, because collected user feedback is not typically positive.

A summative survey can be conducted to ensure the usability of such input devices. Researchers often use questionnaires to find the usability of commercial off-the-shelf input devices [3, 7]. These questionnaires have adapted a Likert scale and can provide beneficial assessment if tailored to the training situation correctly [6]. The scale can follow either an odd- or even-numbered system, though it has been found that an odd-numbered scale prevents the halo effect, limits impairing validity, and provides appropriate balance for sensitivity and efficiency [8–10]. Additional research recommends a 7-point Likert scale due to optimal reliability [11–13]. For this research initiative, one set of questions contained a 5-point scale (ranging from novice (1) to expert (5)) and was modified to assess user experience with regards to game genre and controller type. Similarly, the usability rating for the controller consisted of a 7-point Likert-scale (ranging from strongly disagree (1) to strongly agree (7)). Additionally, free response questions were integrated to gain meaningful insight through additional coverage of controller usage.

In effort to assess the reliability of the questionnaire, Cronbach’s alpha was used to determine the effectiveness of the questions. To assemble the usability questionnaire, an in-depth literature of usability questions (including topics such as usefulness, satisfaction, and ease of use of the video game controllers) were compiled to determine which were most applicable. With permission from the source authors [3, 7, 14, 15], the questions were modified to assess the usability of the VW navigation devices. The subscales that were incorporated included aspects of Usefulness, Ease of Use, Ease of Learning, Satisfaction, Device Comfort, and Overall Device Satisfaction. The definitions used are according to the original sources: Usefulness refers to a device’s capability to accomplish a task efficiently; Ease of Use focuses on the simplicity of a device, and denotes the level of effort and difficulty when utilizing a device; Ease of Learning applies to comprehending how the device works and to adopting the skills required to use the device; Satisfaction determines whether the device is suitable for the virtual simulation; Device Comfort refers to operating the device by assessing levels of fatigue and comfort; and Overall Device Satisfaction identifies whether the device appeared to be difficult or easy to use for completing the task.

The purpose of this usability research initiative is twofold. First, the usability questionnaire was developed to assess the reliability of the questions. Second, the data collected from the NCOs was evaluated to assess the effectiveness of the VW navigation devices used for training.

2 Method

2.1 Participants

Fifty U.S. Army Soldiers, in the rank of Corporal or Sergeant, served as participants in the data collection events. Each Soldier was a student enrolled in the U.S. Army’s Warrior Leader Course at Camp Blanding Joint Training Center, Florida. The aggregated sample was composed of 37 males and 13 females. All individuals were U.S. citizens, between the ages of 18 and 40 (M = 27.02, SD = 4.77), with normal or corrected-to-normal vision, and full color vision. No compensation, in any manner, was provided for participation.

2.2 Survey Design

Within the VW training, Soldiers were asked to complete four collective tasks (i.e., reacting to contact while dismounted, to a near ambush, to a far ambush, and to an improvised explosive device (IED)) through the Army’s VBS3 software, while being supervised by the Small Group Leaders (SGLs). The usability questionnaire assessed the reliability of its subscales as well as documented the Soldier’s reaction to the virtual simulation, in reference to using a standard desktop with a keyboard and mouse.

2.3 Measure

The usability questionnaire was a 44-question individual-level survey that asked participants to subjectively rate their gaming experience, their experience with the simulation, their interaction with the simulation’s game controller, their interaction with other interface devices, and their overall response to the VW training scenario. The questionnaire was recreated from previously published works that examined the type of video game controller, controller use, satisfaction, gaming experience, and user interface experience [3, 7, 14, 15]. The majority of questions followed a 7-point Likert scale, with three questions reserved for free-text response. Additionally, the questionnaire consisted of several items that were reverse-scale scored.

2.4 Procedure

The data was conducted at the Florida Army National Guard’s 211th Regional Training Institute (Camp Blanding, Florida). At the Army training facility, each Soldier was pre-assigned to a squad (46, 47, and 48 respectively) and then asked to read and sign the consent form. Following the consent forms, all three squads were transported to the computer lab (on-site) where under the direction and guidance of the SGL’s to complete the four previously mentioned tasks using VBS3. After completing the virtual simulation training, the experimenter administered the usability questionnaire and then provided each participant with a debriefing form.

3 Results

In order to obtain meaningful results, a minimum of 30 participants was needed in accordance with the behavioral science sample size standard [16]. Data collection efforts revealed a sample of fifty participants who completed the usability questionnaire. The questionnaire was evaluated to determine the reliability of the subscales including Usefulness, Ease of Use, Ease of Learning, and Satisfaction. Additionally, the usability data was analyzed using statistical means to explain participant’s Game Experience with several different interfaces, as well as experience with genre types, including First Person Shooter, Massively Multiplayer Online, and Virtual World. Finally, an examination of each participant’s free responses regarding the overall interaction with the system was logged to add insight through anecdotal evidence.

3.1 Reliability of the Questionnaire Subscales

Usefulness.

To assess the internal consistency reliability of the Likert-type subscales created for the usability questionnaire, the Cronbach Alpha was calculated and reported. The first subscale focused on questions examining Usefulness. The Statistical Package for the Social Sciences (S.P.S.S.) output for the three items revealed a Cronbach’s alpha = .77 A closer look at the Corrected Item-Total Correlation (Table 1) revealed all three items had a correlation greater than .5, which established a reliable correlation between each item and the total. Additionally, in instances where there are few items listed on a scale, the inter-item correlation is also reported. In this case, the mean inter-item correlation was .53, with values ranging from .41 to .73. This suggests a strong relationship among the items, furthering the support for validation of the scale.

Table 1. Corrected Item-Total Correlation for Usefulness
Full size table

Ease of Use.

The next subscale examined for internal consistency was Ease of Use. The Cronbach’s alpha reported for all 11 items = .93. The Corrected Item-Total Correlation revealed all items had a strong correlation (> .5) between each item and the total score from the questionnaire. The mean inter-item correlation value reported was .55, with a range from .21 to .91 (Table 2). The results suggest a strong relationship among the items.

Table 2. Corrected Item-Total Correlation for Ease of Use
Full size table

Ease of Learning.

Following the analysis of the Ease of Use subscale, Ease of Learning was examined for internal reliability. The results suggested a Cronbach’s alpha value = .81 for all three items. The Corrected Item-Total Correlation revealed that the correlation between each item and total score had a strong correlation (> .5) (Table 3). Furthermore, the mean inter-item correlation value reported was .59, and ranged from .52 to .67. Overall, the results suggest a strong relationship amongst the items that contributed to the Ease of Learning subscale.

Table 3. Corrected Item-Total Correlation for Ease of Learning
Full size table

Satisfaction.

The final subscale assessed for measuring reliability was Satisfaction. The Satisfaction subscale revealed a Cronbach’s alpha value = .71. The Corrected Item-Total Correlation showed a relatively strong correlation between each item and the total score (reported values > .5) (Table 4). Despite the small number of items, the mean inter-item correlation was also reported as .56, which suggests a strong relationship between the items (despite that the Cronbach’s alpha if item deleted reported no item value).

Table 4. Corrected Item-Total Correlation for Usefulness
Full size table

Game Experience.

Part I of the Game Experience section of the usability questionnaire asked participants to indicate their experience with seven different interfaces, or controller inputs. The focus of the Game Experience section was essential to understanding the level of familiarity a participant had with a specific interface. Each participant rated their user experience with each device, ranging from novice to expert. Included in the usability questionnaire was the use of a standard desktop keyboard and mouse; devices with consoles, and a pad. Preliminary analysis from the questionnaire indicated that participants were most experienced with a keyboard and mouse, and least experienced with a joystick. Equal experience was found between the steering wheel and pad. Table 5 illustrates the controller type and the averages for the participants’ usage.

Table 5. Averages for Controller Types
Full size table

Part II of the Game Experience section of the usability questionnaire asked participants to indicate their experience with eight game genres. The eight genre types included First Person Shooter, Massively Multiplayer Online, Virtual World, Strategy, Role Playing Games, Simulation, Puzzles, and Racing. The answers revealed which genre a participant had the most experience with, in order to get deeper insight into a participant’s likely gaming preference or interaction style. Each participant rated each genre from novice to expert. Preliminary analysis of the means indicated that the participants had the most experience with racing games, and equal experience among the First Person Shooter and Puzzle genres. The least amount of experience appeared to be in the Virtual World Genre. Table 6 shows each genre and the averages for genre experience. Interestingly, a preliminary analysis of the highest mean controller type and the highest mean genre experience indicated an unexpected pairing of the keyboard and mouse with the racing genre.

Table 6. Averages for Genre Experience
Full size table

Free Response.

The Free Response section of the usability questionnaire allowed participants to describe their overall interaction with the system. All 50 individual were given an opportunity to pinpoint specific issues faced while conducting the training task in the VW. Approximately 10 % of the participants indicated a positive interaction with the VW using the keyboard and mouse system. Alternatively, 14 % described the keyboard and mouse as disruptive and the overall desktop computer was limited making it difficult to use. With regards to meeting expectations, 16 % indicated the keyboard and mouse controls was favorable and functional. On the other hand, 18 % viewed the controls with increase difficultly and not easy to use. Interestingly, four participants explicitly stated that an Xbox control would be a preferred choice. The last free response question gauged the overall impression of the system. Approximately 34 % of the participants view the overall system as useful and a good learning tool while 12 % felt that the system was ineffective and needed more time to familiarize themselves with the keyboard and mouse. Finally, eight participants recommended incorporating a video game controller or game pad to enhance the overall training experience.

4 Discussion

The research initiative examined a usability survey in the aim of determining the appropriate controller for virtual world training, and determining the reliability of the usability questionnaire. Several subscales resulted in a high Cronbach’s alpha, including Usefulness, Ease of Use, Ease of Learning, and Satisfaction. The Ease of Use subscale signaled a valuable relation: the subscale had the highest Cronbach’s alpha (.93) and the most questions (a total of 11). This indicates that the variation in questions may be enough to satisfy the subscale [17]. Even though the subsequent subscales had between two and three questions, the Cronbach’s alpha combined for each subscale was above .70. The questions may have been phrased appropriately to gather useful information [18].

The results showed that the Device Comfort and Overall Device Satisfaction subscales were not statistically significant. Specifically, the Overall Device Satisfaction subscale included a single statement for completing (i.e., “Overall input device was…”) which could have been a source of low reliability for the subscale. Gliem and Gliem [17] noted that unreliable measurements may occur with individual scores. Consolidating the Overall Device Satisfaction subscale with the Satisfaction subscale is posited as a potential method to increase reliability. Further, the subscale of Device Comfort failed to provide significant results, which could be due the limited amount of time provided within the system. The Device Comfort section asked participants about fatigue of the wrist, arm, and fingers. However, the time spent in the simulation was not sufficient for the onset of such symptoms. Fagarasanu and Kumar [19] report that repetitiveness is a factor that contributes to carpal tunnel syndrome. The time participants spent was limited to a total of four hours, including a ten-minute break. Thus, this subscale may be better suited for a longitudinal study or training involving a longer time in the simulation.

Lastly, the 7-point Likert scale forms were presented to users with an additional Not Applicable (N/A) option. Throughout the course of refining the survey, the need for this option was defeated by generalizing the questions, affording multiple input types for inclusion. For instance, the original survey pertained to providing two sections of similar questions that participants answered based on the controller used (e.g., keyboard/mouse vs. game controller). However, having generalized the questions, the N/A seemed to be used by participants to avoid questions. The University of Washington Office of Educational Assessment [20] recommends using N/A if appropriate, but sparingly for opinion items. Further, the time allotted for as much training as possible, so time taken for filling out the survey information was limited. This may have promoted the use of the N/A option. Future studies will change the N/A option to only include the 7-point Likert scale within the questionnaire, in order to retain a higher sense of control.

Additionally, the usability data was analyzed using statistical means to explain participant’s game experience with several different interfaces, as well as experience with genre types. Specifically, the highest means from Tables 5 and 6 indicate that the racing genre and the keyboard/mouse interface have the highest controller and genre experience levels, respectively, among the participants. This is an unexpected discovery. Previous studies [21, 22] show that the realism of a steering wheel and pedal interface used in a racing game enhances the overall gaming experience. However, these previous studies did not account for the economy of using an interface that comes “standard” with a console or computer. Although these more realistic interfaces may indeed enhance a user’s gaming experience, the devices are limited in their applications. For example, a steering wheel must be used primarily for a racing genre and cannot be effectively or conveniently mapped to another game genre, such as a first person shooter or a puzzle game. The utility of a keyboard and mouse or console controller allows the user to apply the interface to multiple genres, without requiring the gamer to use additional financial resources for perceived realism. The user can also customize the control settings to suit his or her process of gameplay. It may be implied that the utility of certain devices, such as the keyboard and mouse or console controller, allows the user to fluidly map device control settings across multiple genres without having to arduously re-learn a different interface for each game genre.

Further, game controller implementation shows advantageous traits, in comparison to traditional desktop setups, in several ways. Game controllers such as the Xbox have a smaller design that affords the user the ability to maintain both hands in close proximity on one device, unlike a keyboard and mouse which requires two hands that may be a substantial distance apart. The increased distance may add to the difficulty experienced by the user, especially when moving. This would likely be observed when a task or action requires the combined use of the mouse with a button on the far left side of the keyboard. With a game controller, the use of joysticks close to each other reduce the difficulty associated with movements. The keyboard has an additional disadvantage: there are many keys that are not utilized, yet are still able to be pressed, therefore potentially disrupting the user’s flow of actions within the game. A game controller has a limited amount of buttons with each serving a specific purpose. This purposeful advantage coupled with a smaller design gives the user an ability to quickly sequence multiple buttons to manipulate, move, and interact within the game when needed. Overall, a more user-friendly game controller, such as the Xbox controller, will likely reduce the amount of time the user needs to become oriented to the controls. The less time that is needed for practice, the more time the user is able to focus on achieving the desired training effects.

5 Conclusion

The preliminary results serve as a foundation for future research on the usability of VW navigation devices. Modifications to the usability questionnaire will include removing the N/A option that coincided with the Likert scale, adjusting the future length of training time to legitimize the Device Comfort subscale, and combining the Overall Device subscale with the Satisfaction subscale. Finally, future research initiatives will seek to incorporate additional types of controllers to usability testing, as this paper did not exhaust all interfaces (e.g., motion-sensors for body-based interactions, touchscreens, and trackballs). The objective of incorporating these adjustments will be to confirm the reliability of the questionnaire, discover questionnaire aspects that need to be augmented, and reveal interfaces beneficial for VW tasks. Although this questionnaire is currently focused on gaming controllers for VWs, future testing could expand to other applications.