Keywords

1 Introduction

1.1 Motivation

Even before the 21st century, humans have made use of computer-aided systems that can relieve them of certain tasks. The application ranges from simple cues to automated assistance systems [1]. Especially in the 21st century, Augmented Reality (AR) technology has become increasingly important in the industry, research and military domains [2].

On modern battlefields, AR has the advantage that information of any kind can be presented directly to the crew, which can significantly assist in the understanding of the situation. Using this approach, the crew can keep up to date with the tactical situation while also performing other tasks.

The use of AR can also bring tactical and operational advantages while coordinating multinational efforts, such as within the NATO. Different nations with different languages can share the same repertoire of AR elements and develop a common understanding.

However, the wrong use of AR can also have disadvantages, e.g. if the information is not adequately provided, it can lead to cognitive overload. If the cognitive workload is too high, the crew tends to postpone one or more low priority tasks to a later time (workload debt). If these postponed tasks reach a certain amount, the crew can be more stressed and the failure rate will increase (workload debt cascade) [3]. In addition, if used incorrectly, it can lead to inattentional blindness [4].

In order to ensure the practical use of AR in different scenarios, technologies, humans and AR cues must interact in symbiosis with each other. Therefore, a methodology, Use Cases and an explorative field trial were designed. As the design and use space of human & AR systems is rather large and only partially investigated with experiments, an explorative approach with participation of potential users is more effective to come to working prototypes. An overview of the exploration paradigm can be found in [5], a description of its instantiation as an Exploroscope is provided in [6].

1.2 Methodology and Background

Schwarz and Fuchs [7, 8] present how assistance systems can adapt to the human state using multiple measurements to determine multiple dimensions of the human state, such as situation awareness, attention, fatigue, emotional state, workload and motivation. Depending on the human state, assistance systems can support the human more or less to fulfill their tasks. An option to bi-directionally adapt human and automation behavior are patterns (e.g. [9, 10]) and more specifically interaction patterns (e.g. [11,12,13,14,15]).

Specifically, situational awareness (SA) is a decisive aspect of survivability for combat vehicles and soldiers in the battlefield. The introduction of optical, radio and acoustic sensors for monitoring the battlefield, and in particular the fielding of Battlefield Management Systems (BMS) which provide an overview of the tactical situation, has significantly improved the situational awareness for combat vehicle crews. However, in some cases, the information in the BMS is displayed separately from other sensor feeds, e.g. periscope, leaving the crew to fuse these two key sources of information. In time-critical situations, it is difficult to fully exploit the information provided by the BMS and other information systems, since crew members have to concentrate on the scene where the action is taking place. Ideally, the tactical information in the BMS should be converted and mapped onto the operator’s field of view, e.g. by using a heads-up-display, regardless of his role. AR can be used for overcoming these deficiencies by presenting the information from the BMS, directly in the operator’s sight, typically in the form of graphical symbols. Visual information may be further augmented by acoustic and haptic cues, e.g. audio alerts or vibrations. Supported by this AR functionality, the crewmembers can pay full attention to what is going on in the vehicle’s area of operation, while at the same time stay updated on the tactical situation within the sights’ field of view.

The amount of information being displayed to the crew should depend on the current situation (e.g. planning of the route, driving to a certain location, engagement with hostiles, etc.), the current human state (stressed, tired, bored, etc.) and the current quality and availability of information and assistance capabilities. AR can be used to adapt human behavior, e.g. to move the current focus to objects that might be important and to implement interaction patterns which can be helpful to structure the mechanism of behavior adaptation of the human or other assistant systems, e.g. present information of the BMS depending on the human’s actions or state.

1.3 bHSI and Interaction Patterns

As can be the case with any changing system, the stability balance present in a system of resting state can quickly become unstable through external factors. This can also happen in an adaptive Human-Machine System if the interaction is not well designed and robust, resulting in a partial state of confusion. E.g. in case of an emergency, the control of a vehicle can be shifted from the less capable actor to the more capable one without understanding of the reasons for such control shift. This results in a subsequent reduction of situation awareness. Interaction patterns can be used to design how interaction happens and therefore, maintain a well-balanced system. In the case of Augmented Reality, interaction patterns are an option to determine which and when a different set of cues should be used in order to adjust to a changing situation. Interaction patterns discussed by [11] and [12] use an escalation scheme based on the severity of the situation. This is similar to a case encountered during military operations, when the crew needs a different type of augmentation depending on patrol or direct combat situations. With the present methodology, it is possible to determine the parameters and extent of the augmentation required in each given scenario. While the trigger for the change of the level of augmentation was not further investigated in these trials, the tools developed during this project can significantly help to identify the appropriate level of augmentation necessary to increase the situational awareness. The tool presented (exploration sandbox) makes it possible to achieve different configurations of each AR cue by changing their properties (e.g. color, brightness, size, etc.) and cooperatively and directly incorporate the user into the design process.

2 Use Cases

Use Cases describe generically what can happen between the actors of a system to achieve a goal of the primary actor. Actors are something or someone that exists outside the technical system under study, but is part of the system under study and under design. Actors take part in a sequence of activities in a dialogue with the technical subsystems to achieve a certain goal. They may be end users, other systems, or hardware devices. Normally, Use Cases describe abstract use situations.

The abstract form of Use Cases facilitates a common understanding between users (commander, gunner, driver, etc.), designers and developers of different faculties (engineers, psychologists, etc.) without going into technical details.

Figure 1 shows a representation of the interaction between two combat vehicle systems (subsystems), each depicted as a human machine system, which includes their respective soldiers interacting with the BMS in a combat vehicle.

Fig. 1.
figure 1

System diagram of two combat vehicles with their crew and Battle Management System (BMS) [NATO AVT-RTG 290] based on [16].

Full size image

An urban scenario can have multiple Use Cases. The scenario in Fig. 2 was developed in a workshop together with soldiers from Norway, the Netherlands, the United Kingdom and Germany as part of the NATO Research Task Group 290 “Standardization of Augmented Reality for improved Situational Awareness and Survivability of Combat Vehicles”.

Fig. 2.
figure 2

Urban scenario and respective Use Cases [NATO AVT-RTG 290]

Full size image

The use case this paper focus is called “Drive to City Border”. In this use case, a specific user need of the commander is to know where the blue team is. The respective user story from a commander’s point of view is: “As a commander, I want to know where friendly troops are, in order to improve coordinated maneuvers and prevent fratricide”. A respective interaction pattern to address blue team tracking in terms of form, color, symbology, etc. was explored, keeping in mind the adaptation of the presented information of the situation, the user state, the quality and information availability.

3 Preparations for Explorative Study

For the exploration of AR cues, two different approaches were used. The first one involves creating a virtual environment to simulate a vehicle, Use Cases and AR cues. The second approach requires a more real world implementation using a test vehicle and positioning sensors. Each approach presents some limitations described more in detail below.

3.1 Laboratory Trials

The virtual approach uses a virtual map where the participant interacts and drives a vehicle through VR glasses (HTC Vive Pro Eye). The map is designed to resemble an urban scenario. It is possible to present to the user not only visual cues, but haptic and auditory cues as well. For the visual cues, the NATO Symbols for Friendly, Neutral, Hostile and Unknown were used. Each cue can be adapted to the user preference in regards to the distance to the object (visibility radius; Fig. 3 left). Additionally, it is possible to mark specific areas with different colors and transparency levels. Other cues can be done through haptic and acoustic modalities. In the acoustic part of the Exploration Sandbox, tones of different frequency, volume and repeatability can be designed. These are played through headphones to the user. Haptic cues can also be implemented through haptic elements. They can reproduce different vibration patterns that vary in frequency, strength and position.

Fig. 3.
figure 3

Left: Map shows the placement of objects and their visibility radiuses; Right: participants point of view in the simulation (Color figure online)

Full size image

Since the main objective is to increase situation awareness based on the criticality of the situation, a dynamic change of the amount of information presented to the user is necessary. This is achieved by using an interaction pattern with an escalation scheme (Fig. 4). The level of criticality is determined by the distance of the ego-vehicle to an object of interest. Each escalation step uses a specific combination of haptic, visual and acoustic cues. For the laboratory trials, different parameters and cue combinations were used for each object of interest (hostile, friendly, unknown, neutral) depending of their level of threat to the ego-vehicle.

Fig. 4.
figure 4

Example of a distance-based escalation scheme for a hostile object (based on [13])

Full size image

The activation distances as well as the specific parameters and combination of cues for each escalation step were explored using an Exploration Sandbox (see Sect. 3.3). It is important to mention that participants were involved in the definition of these parameters, based on personal preference and experience.

The first approach takes place in a virtual environment. The second approach, which takes place in the real world, uses a vehicle in combination with AR glasses to drive in the real environment.

3.2 Field Trials

One of the advantages of using a real-world approach is that some of the possible problems encountered when using VR technology can be avoided, e.g. simulation sickness. Nevertheless, this kind of approach requires a more complex setting and in some cases, costly resources which may not be available.

In this field trial, the user sat on a static IFV (Infantry Fighting Vehicle) PUMA, which added realism to the scenario. Symbols, distances and other elements were presented through AR glasses (DreamWorld DreamGlass). Every object of interest was equipped with its own GPS system. This data was used to correctly display the AR symbols in the DreamGlass. Further information on this object, such as the distance to it, can also be displayed (Fig. 5).

Fig. 5.
figure 5

Image from point of view with the AR glasses; Top: NATO symbology and distances; Bottom: alternative view with frame around object (Color figure online)

Full size image

3.3 Exploration Sandbox

The exploration sandbox incorporates an extensive range of properties for each modality that can be customized to create many different versions of each AR element (Fig. 6). This allows designers, users and stakeholders to participate dynamically, directly and quickly in the design process and to design behavioral patterns. The created AR elements can be displayed in a combination of visual, haptic and acoustic forms, e.g. through AR or Virtual Reality (VR) glasses, a haptic vest or speakers/headphones, respectively. Since the Exploration Sandbox UI is independent from the rest of the system, it can be used in the laboratory or in field trials. This is possible thanks to the middleware ROS (Robot Operating System) that connects hardware, simulation and UI. Because of this modular approach, new interfaces, e.g. haptic controls or other AR glasses can easily be integrated.

Fig. 6.
figure 6

Exploration Sandbox for visual AR cues

Full size image

In this context, different modes of augmentation can also be explored to find the right amount of information for the current situation, e.g. training, planning or combat. The elements defined in the process are intended to be used as a standard or guideline in the future design of AR systems for military and partial civil applications.

4 Field Study

A field study was conducted in order to investigate technical feasibility and usability on the one hand, and the understanding and design of the used symbology on the other hand. The study was done with the participation of potential end-users, soldiers of the German Armed Forces at a military base.

The goals of the study was to (1) evaluate the technical and technological properties and challenges of the system in an ecologically valid context and (2) collect feedback and suggestions on the design of the symbology. In total 10 (9 males, 1 female) military drivers took part, the youngest being 30 and the oldest 57 years old (mean = 44.25, SD = 10.91). The above-mentioned technical setup was placed on site of a military base with direct access to an armored vehicle (IFV PUMA) and availability of the participants.

The study consisted of two parts: first an informal evaluation while being mounted (i.e. in the driver’s seat in the armored vehicle), and afterwards a one-on-one participatory design workshop. For the former, the participants were presented with different situations und scenarios in which at least one and at a maximum three elements were shown through the AR-glasses, one of which was a possible waypoint and two were possible friendly symbols. The augmentations could be either static or dynamic, within or out of the field of view, and occluded or open. Of the two friendly symbols, only the first one (Friendly 1) had varying properties, the second one (Friendly 2) was always static, in the field of view and not occluded. For a more detailed description of the scenarios, refer to Table 1.

Table 1. Different scenarios of the first study, only the properties of the symbol ‘Friendly 1’ were altered
Full size table

Note that part of the scenarios were completed with “closed hatch”, meaning that the participant was in the inside of the tank as opposed to “open hatch” when the driver’s head sticks out of the vehicle. After the situations with varying and possibly dynamic augmentations were completed, situation two (open hatch, two friendly symbols and one way point) was held constant and the properties of the symbol were altered according to the following:

  • Change in transparency (0%, 30%, and 70%)

  • Fluorescent green vs. NATO color scheme (in this case blue)

  • Distance in corner of symbol vs. distance inside the symbol

  • Framing of the object and symbol as explanation above vs. symbol directly on object

In order to get as much ad-hoc input as possible, the participants were specifically encouraged to follow the think-aloud protocol [17], especially when they were having issues or concerns. However, since they had no previous experience with this technique, nor was there enough time to extensively instruct them on how to do this, the researcher was interviewing them during the trial about the following themes:

  • Understanding of the presented symbology

  • Understanding of the distance indication

  • Issues regarding understanding

  • Issues regarding visibility

  • Issues regarding technical equipment

  • General evaluation of the system

After finishing the current system state evaluation mounted on the vehicle, the participants were to fill in a 7-point Likert scale indicating their agreement from one (do not agree at all) to seven (completely agree) regarding the following three statements:

  1. 1.

    The system would improve my combat ability.

  2. 2.

    The system is user-friendly.

  3. 3.

    The system would provide me with additional security.

This questionnaire was administered again after the participatory design session evaluating an optimal system with the discussed optimizations implemented. In this session, the participants were shown the different design options that had already been presented to them through the AR glasses and were asked to give their considerations concerning the design of the symbols. It has to be noted that not all had seen the augmentations clearly in the glasses; this made it necessary to present them on a computer screen in order to assess their usefulness and fit.

The way in which the system was implemented and presented to the soldiers while being mounted yielded an average score over all three questions of 4.26 (SD = 1.22), which is very close to the middle of the 7-point Likert scale. For a theoretical ideal implementation this score was 5.44 (SD = 0.82), pointing in a more favorable direction.

Since this was no experimental, validative study, but an explorative one, no inferential statistical analysis was performed. Which also means that possible differences between the two scores are neither proven, nor of particular interest for the goal of this study. The results only indicate that the system in its current state received mixed feedback and it would probably not be advisable to implement it in a vehicle as-is. However, there seems to be potential for use of the technology, even if it was not perceived as overwhelmingly positive. This is indicated by the rather high average score for the theoretical ideal implementation of aforementioned system.

To reach the full potential of the AR system, the collected qualitative data needs to be reviewed in order to identify problems and hindrances, as well as strong points and potential uses. The interviews during and after the trial were transcribed and coded. Topics that were mentioned by more than half of the participants (at least 6) are listed below in descending order:

  • NATO green is better visible than blue

  • No intuitive understanding of the meaning of the used symbol (NATO symbology)

  • Number presented is intuitively understood as distance to target

  • Augmentation does not bother while driving with closed hatch

  • Augmentations hardly visible (open hatch)

  • Augmentations are blurry (open hatch)

  • Augmentations are more strongly and clearly visible with closed hatch

  • Wrong perception of the color (blue is identified as white, open hatch)

  • Double vision of the augmentations

The visibility of augmentations in AR glasses is strongly influenced by the amount of background light [18]. Half of the trials were conducted on a day that could be described as sunny and therefore was relatively bright, as opposed to the following five runs that were completed in cloudy weather, which resulted in the perception of a higher saturation of the augmentations. For this reason, the symbols were visible fundamentally better with a closed hatch because the interior of the vehicle is comparatively dark.

During the participatory design workshops, almost all participants (eight out of ten) determined that the framing design (see Sect. 3) was best suited for the use in an armored vehicle. This was mainly because it had the smallest risk of occluding important information and was least obtrusive. However, there was less consensus about the coloring of the symbol, with some participants preferring the NATO green and others favoring using the corresponding color for the identified object (in this case blue for friendly). This could have been influenced by the weather, respectively light conditions, in which the trials were run. The green symbols were more easily seen in the bright conditions and therefore could have been chosen over the different coloring by participants on this day. Another design consideration that was mentioned by some participants was the display of multiple objects in one location and how to prevent overlapping and therefore ultimately loss of information or information overload. A suggestion by one participant was to extend the frame so that it covered all relevant objects that are in proximity and then displaying a number on how many there were. Solving this issue merits further investigation and needs to be addressed in a follow-up study.

5 Outlook

The results of the exploration provide a discussion baseline for the use of AR cues and propose the use of interaction patterns as an adaptation strategy. The explorative process with the dynamic creation tool can be used to fine-tune the system to find the best combination of cues, properties and activation of each escalation step.

During the field trials, it was clear that there is room for improvement, especially in the GPS communication system, which proved to be inaccurate at times. An alternative for this is to use not only GPS but also other location acquisition systems, e.g. Lidar. This would provide the opportunity to gather extra data about the shape and orientation of objects, and therefore, further classify objects of interest.

Since not all objects represent the same level of threat to the ego-vehicle, even if they are of the same type (e.g. hostile IFV vs. hostile scout soldier), the combat ability of the object of interest in combination to the capabilities of the ego-vehicle can be used in the design of the interaction patterns. Interaction patterns could then be dynamically adjusted depending on the previous object classification, environment and other aspects of the use case. Subsequent changes to the levels of augmentation can also be influenced by this.

When analyzing the interaction between the user and the augmentation and following comments from the explorative workshops, it became clear that the user should remain capable of changing the augmentation or turn it off completely in situations of high stress or according to its personal demand or preference. This could be achieved by incorporating other technologies such as gesture/gaze-based interaction that can add other layer of interaction possibilities by using free resources directly in the line of sight of the user.