Workload perception in drone flight training simulators

Highlights

• Association exists between Workload Perception and performance in drone flight simulator tasks.

• AWT a NASA TLX adaptation test showed high reliability, moderate sensitivity and predictive value.

• Mental Demand had higher correlations with errors and time delay during flight simulation tasks.

Abstract

Workload perception was measured in a drone flight training Simulator computerized situation. There has been increasing research in recent years on the topic of Remotely piloted aircrafts (RPA). Eleven participants were tested for workload perception during a drone flight simulator training. Reliability, sensitivity and correlations were studied for the workload scale and its relationship with the simulator training tasks. Overall, there were clear effects of mental demand as showed in the workload perception during the training tasks. Reliability for the workload scale showed good score and sensitivity showed mental demand as the most important factor compared to the other parameters measured obtaining highest correlations with landing tasks and number of errors. In our results, we have seen how the AWT (adapted from NASA-TLX) showed good sensitivity in assessing the mental burden of participants. In our research, participants scoring higher in the mental demand subscale showed greater difficulty finishing training tasks, and also showed longer time delays in performing both training sections of the simulation. These types of tools measuring workload perception and virtual training systems can be used in future research, to see how this cognitive aspect affects piloting skills and its possible safety and training implications.

Introduction

Unmanned aerial vehicles (UAV), also known as remotely piloted aircrafts (RPA) or more commonly as drones, were initially developed for military purposes. After World War II, there were several countries working on this technology, in order to conduct surveillance without being seen by the enemy and risking human lives.

Until recently, drones had been limited to the military sphere, but nowadays, technological improvements, advances in communications, and battery technology, have ensured that small, low-cost UAVs allow civilians to work and conduct experiments with drones.

These aerial robots are a good solution because they can cover a wide area without touching the ground. Therefore, they can be used to explore, for example, the remains after a catastrophe (Astuti, Longo, Melita, Muscato, & Orlando, 2008). Their high mobility, the possibility of use in environments that are dangerous to humans (Kontitsis, Tsourveloudis, & Valavanis, 2003), and their ability to reduce operating time and improve the identification of causes and effects of crises, make them a useful tool for various types of tasks. These tasks include search and rescue missions using high definition imaging and thermal imaging (Rathinam et al., 2007); analysis of the gas composition within volcanoes (Astuti et al., 2008); surveillance operations including inspection and monitoring of the boundaries of rivers, bridges, and shorelines (Rathinam et al., 2007); fire monitoring in forests (Casbeer, Beard, McLain, Li, & Mehra, 2005); search for ground targets in unknown regions (Xie, Ye, Luo, & Li, 2012); and mapping (Templeton, Shim, Geyer, & Sastry, 2007) among other utilities. The still-expanding civil applications offer a multitude of solutions: review of high-voltage wiring, agriculture, mapping, measuring structures, anthropology, etc.

There has been increasing research in recent years on these small aerial vehicles, most of it with the object of interest being the use of algorithms or hardware specifications to make the drone capable of autonomous operation, or performing different types of tasks more effectively. This approach forgets that although there is no pilot on board these aircrafts, they require significant human interaction.

The fact that there is no human in the vehicle is misinterpreted by some as there is no human in the system; however, RPAs are complex systems that require a lot of human involvement and they involve mixed human/robot systems to an extent (Kontitsis et al., 2003). Consequently, the study of human factors related to drone piloting or quadcopter operations may significantly contribute to the performance of tasks.

In literature regarding human factors in RPAs, a bulk of the existing research is based on UAV for military use, excluding the study of variables that can affect the operator of a small quadcopter. Similarly, state air safety agencies are increasingly setting rules on the use of these vehicles for safety and security reasons. This is a clear sign of the importance of the fact that we need to control the use of these vehicles in the airspace, as well as the importance of adequate training for future drivers, where issues related to human factors are included.

Research has shown that while the percentage of aircraft accidents attributed to mechanical failures has decreased dramatically over the past 40 years, the percentage attributable, at least in part, to human error has dropped to an even lower percentage (Shappel & Wiegmann, 2000). This situation shows the need to focus more on working with humans, and the need for people to remain indispensable in the use of UAVs. “Pilot error” is often the reason given for an aircraft accident; however, human error usually has an underlying cause. These causes can include high (or low) workload, fatigue, and poor knowledge of the situation or inadequate training among other causes of which one or some can slow performance and lead to an accident or failure of the objective (Manning, Rash, LeDuc, Noback, & McKeon, 2004).

While automation is being increasingly used in the working of these devices, we cannot forget that automation can also increase the workload of the operator and reduce situational awareness (Ruff, Narayanan, & Draper, 2002). Similarly, high levels of automation can also prevent the operator from quickly intervening to override automation if necessary (McCarley & Wickens, 2005). It is, therefore, very important to note that the automation of various functions should not eliminate human intervention in full (Hopcroft, Burchat, & Vince, 2006).

In human factors research, it appears that complex tasks are performed most successfully when the system is designed to support the needs of human beings instead of removing the human from the system (Abbott, Slotte, & Stimson, 1996]. In many cases, the goal of eliminating the human from the system has led to major system failures, specifically because the system was not designed to support interaction with the human (Casey, 1998). The combination of the strengths of humans and robots to achieve a cooperative task is becoming a popular paradigm (Bruemmer et al., 2007, Crandall and Cummings, 2007, Fong et al., 2003). Adjusting the autonomy levels of the robot to allow human input is a good way to achieve an optimal combination in mixed human/robot teams. The underlying assumption is that the robot performance increases with more human input (Kaupp & Makarenko, 2008). If the human operator is an indispensable factor when working with UAVs, it would be essential to study variables that affect human performance, specifically in tasks where an operator interacts with a rotary UAV, in order to leverage the full potential of the operator to perform the task in the best possible way or even to prevent possible accidents or failures.

Human performance is affected by a variety of influences in both internal and environmental tasks. The functions that modulate human performance are equations derived from empirical data that are used to determine how human performance is affected by the combination and influence of factors found in specific conditions. Examples of these modulating functions are sleep quality and quantity, ambient temperature, stress, and workload (Aasman, Mulder, & Mulder, 1987).

Workload is defined as the combination of the demand for labor and the human response to this demand (Mouloua, Gilson, Kring, & Hancock, 2001). The assessment of workload is a key point in the research and development of systems for human-machine communication in order to ensure the safety, health, comfort, efficiency, and long-term success of the operator (Rubio, Díaz, Martín, & Puente, 2004). Workload levels vary considerably between extended periods of low workload and intense periods of high workload. An effective work design or schedule usually aims to avoid extremes of high or low demand, which can be a threat to the maintenance of skills (Sauer, Wastell, & Hockey, 1996). On the other hand, prolonged periods of high workload may result in reduced attention, increased stress, fatigue, reduced flexibility, and information processing deficits (Connors et al., 1985, Hockey, 1993).

Continuous periods of high workload increase fatigue, especially after multiple periods of total loss of sleep, long periods of sleepiness, or sleep fragmentation. This degrades the performance, productivity, safety, and effectiveness of the mission. Moreover, this loss of sleep combined with high workload reduces reaction time and decreases alertness (Kmeger, 1999). High physical and mental demands can also cause more errors due to increased fatigue and loss of concentration (Schuetz et al., 2008).

There are a number of tools to assess and predict mental workload. Most of these methods are divided into the following categories: (a) measures based on performance, (b) subjective measures, and (c) physiological measures (Meshkati, Hancock, & Rahimi, 1992). Of these, subjective measures are becoming increasingly important as assessment tools and have been widely used to assess the workload of the operator. The reasons for the frequent use of subjective methods include practical advantages (ease of implementation, no intrusion) and ongoing data, which support the ability of subjective methods to provide sensitive measures of the mental load of the operator (Rubio et al., 2004). There are several subjective tools for measuring mental workload; the most commonly used are the Cooper-Harper Scale, the Bedford Scale (Cooke and Mesa, 2006, Cooper and Harper, 1969, Roscoe, 1987, Roscoe and Ellis, 1990), the Subjective Workload Assessment Technique (SWAT; Reid & Nygren, 1988), and the NASA-Tax Load Index (NASA-TLX; Hart & Staveland, 1988). The information provided by these subjective scales can be a valuable source of information on mental workload in two fundamental ways. First, they can be used to identify specific sources of demand that a specific task may have. Second, they can reveal differences in workload between two or more individuals (Yurko, Scerbo, Prabhu, Acker, & Stefanidis, 2010).

For this study, we decided to use an adapted version of the NASA-TLX. We took into account the facts that the psychometric properties of this test are well documented, the test has been validated, and it has been previously used by the AMES Human Performance Research Laboratory Group at NASA as a tool for the subjective assessment of individual workload in real flights, flight simulation tasks (Battiste and Bortolussi, 1988, Corwin, 1989, Nataupsky and Abbott, 1987, Nygren, 1991, Shively et al., 1987, Tsang and Johnson, 1989, Tsang and Velazquez, 1996), and even unmanned vehicles (Byers, Bittner, Hill, Zaklad, & Christ, 1988). NASA-TLX has greater sensitivity compared to the computer versions of other scales (Hill et al., 1992) and has the ability to modify language and adjust questions to suit specific tasks and needs (Cao, Chintamani, Pandya, & Ellis, 2009).

Evidence suggests that human error is a major contributing factor to accidents in commercial aviation (Wiegmann & Shappell, 2001). Common errors that are also applicable to drone pilots include not initiating the appropriate maneuvers, failing to notice visual and auditory alerts, being unable to maintain good situation awareness, and poor decision-making. Identifying the cognitive factors and underlying neural circuitries that are predictive of pilot errors is a great challenge, as flying is a complex but necessary activity that takes place in a rapidly changing and uncertain environment. According to Hardy and Parasuraman (1997), a pilot's flying performance is dependent on domain-independent knowledge (e.g., cognitive functions), domain-dependent knowledge (e.g., procedural knowledge), pilot stressors (e.g., adverse weather conditions), and pilot characteristics (e.g., age, expertise). Numerous studies have linked flight functioning and cognitive performance, and different measurements of cognitive efficiency have been identified as crucial to piloting ability, including timesharing (Tsang & Shaner, 1998), speed of processing (Taylor, Yesavage, Morrow, Dolhert, Brooks, & Poon, 1994), attention (Knapp & Johnson, 1996), and problem solving (Wiggins & O’Hare, 1995). The notion that specific aspects of cognition play a crucial role in the chain of events leading up to drone crashes suggests that the implementation of efficient screening of cognitive procedures for pilot selection, training, and the development of monitoring and instrumentation systems fitted for the human brain may decrease accident rates and maximize success.