Keywords

1 Introduction

Flying safely is a goal that humans pursue constantly, flight activities refer to highly dynamic operating processes. Although events that threaten aviation safety have decreased yearly because of equipment factors, accidents caused by human error annually have not shown a decreasing rate [1]. Apparently, human operation is the most critical factor influencing aviation safety. Regardless of military and civilian pilots take responsibilities for aviation safety and flight tasks. Specifically, military pilots often carry out diverse tasks and training in potential risk situations which influence their thinking and judgment abilities and may induce human error. The quality and appropriateness of decisions determine whether aviation safety and tasks can be achieved [2]. Jones and Endsley indicated that among aviation accidents caused by human error, more than 75 % were caused by pilot failure in perception [3]. The messages shown in the cockpit must be identified visually, visual perception ability is critical to situation awareness. Endsley indicated the importance of situation awareness to aeronautical decision making, the Pilots have to make appropriate decisions in a complex operating environment within a limited time [4].Situation awareness comprises three sequential processes, namely perception, comprehension, and projection. Pilots experience the stimuli present (perception) and comprehend the present situation (comprehension). Subsequently, pilots project the situation and make plans in brain (projection) to effectively solve various situations and operate the flight aircraft safely [4]. Thus, visual information plays an extremely critical role in flight operation and performance.

Vision is formed when the central foveal tissue of the perception neuron in the retina receives an object image reflected. To obtain data and understand the surrounding environment, people continually perform eye movement, including fixation, saccade, and pursuit movement indicators. Fixation refers to the process of temporarily fixing the gaze point on a certain area or location and transmitting the obtained visual information to the brain for perception and cognitive behavior [5]. Saccade is an eye movement in which the fixation is changed using muscles around the eyes [6]. The occurrence sequence of fixation and saccade forms scanpath, which facilitates clarifying the eye movement of a person and investigating the attention allocation and cognitive processing of a person [7].

Most visual information is obtained through fixation. The results of previous studies have suggested that the change of fixation points during visual search represents the change of attention. The saccade range reflects the wideness of attention [4, 8]. The duration of fixations can indicate the complicacy of external information and internal information processing. Scanpath can be regarded as an indicator reflecting the attention distribution and cognitive strategy of a person [9]. Previous studies stated experienced pilots attained higher fixation counts and longer fixation durations compared with novice pilots [10].

In the cognitive information processing process of humans, more than 80 % of the information is obtained through vision. Eye movement is an extremely crucial sensory information source during the cognitive process [11]. The selection mechanism of visual attention allocation can be divided into bottom-up and top-down cognitive processes. The selective visual attention mechanism is a key factor determining where a person must gaze in an overall environment. According to previous studies, when information is complex or incomprehensible in concepts, the corresponding eye movement differs; for example, increased fixation duration, reduced saccade distance, and increased frequency of regressions [13]. When a person gazes at a specific region, vision is focused on the region [13, 14].

Eye-movement recording techniques began to develop into maturity in the 1970s, Presently have evolved with increasing accuracy [5]. This study examined the visual tracking state, attention allocation, and operating decision processes of pilots executing air-to-surface task. We applied the eye-tracking system to collect eye-movement data. Flight performance was evaluated by professional flight instructors based on the operating abilities and performance of pilots. In addition, we used the indicator of pupil size to determine whether work load affects flight performance. Ayaz et al. claimed that pupil size variations can be used to evaluate the sensitivity of mental activities and mental workload [15]. When emergency situations occur, mental workload may increase drastically and affect operating performance, thereby deteriorating the original flying capacity. Although technology evolution has enabled most of the operating behavior to be performed by automatic systems, pilots still experience increased mental workload to effectively operate multifunctional and high-performance devices [16, 17]. Thus, understanding the mental workload of pilots and decreasing the effect of mental workload on flight performance can effectively improve flight performance and maintain aviation safety.

2 Methods

Participants. This study were recruited total of 18 participants. All of the participants were men and in-service F-16 pilots with an uncorrected visual acuity of 1.0 or higher. The average age of the participants was 32.28 years (SD = 5.08). The total flying time ranged between 560 and 2,800 h (M = 1253.42, SD = 764.52). The participants have to completed a consent form to participate. While the experiment process, no invasive research instruments were adopted to maintain research ethics.

Equipment and Scene Setting. (1) Eye tracking system: The ASL Series 4000 eye tracker at a weight of 76 g was employed to prevent interrupting operations by the pilots. The system records eye movement, images, and digital data. The collected information were stored in a digital video cassette recorder and computer. The sampling frequency was set at 30 Hz, and the resolution was set at 640 × 480 pixels. (2) Flight simulator: The F-16 fighter simulate various flight situations and provide an interactive and reality training environment for pilots. In addition, an instructor to monitor the real-time operating performance of the pilots and evaluation. During the entire experiment, the lighting in the simulation cockpit was controlled to eliminate the effect of light source brightness on pupil constriction reactions.

Scenario. The scenario was an air-to-surface training course. The initial settings are detailed as follows: (1) Weather: The visibility was 7 nautical miles with static wind. (2) Location: Air-to-surface bombing and gunnery training range, with the downwind leg height of 2,000 ft and 400 nautical miles. The pilots were required to dive in at high speed to the expected bomb release point or firing altitude to avoid surface combatant counterforce. The flight simulation course took approximately 10 s for the key operating stages from flatten, tracking, pick-off, to off. Experimental Procedure. First of all, we explained the purposes and procedures of the experiment to the participants. After the participants entered the simulator, we mounted an eye tracker on them and implemented fixation adjustment. During simulation flights, we simultaneously recorded the eye-movement data of the participants and requested the instructors to evaluate the performance. The entire experiment procedure took approximately 30 min, and all of the participants received the same experiment.

3 Result

The flight manual stated that the execution of a single air-to-surface task was approximately 80 s to 90 s. However, the actual flying duration of pilots was different. Because the eye-trackers collected a substantial amount of data, to accurately analyze the eye-movement variation of pilots, we set the pick-off stage as a reference point. A period of 60 s was allotted for analysis. In addition, analysis was conducted in identical time periods to avoid data confusion from affecting the research accuracy.

Eye-Movement Data Analysis in Air-to-Surface Tasks. In air-to-surface tasks, the pilots expended 39.73 s (66.22 %) on average gazing at the HUD. Thus, we set the HUD as the area of interest and analyzed various eye-movement data. The flight performance was used 30 points as the basis for determining high or low flight performance. In addition, the total flying time of 1,000 h was used as a basis for determining pilots with high or low experience.

We used a t-test to perform difference testing and observed a significant difference in total time in zone (t = -2.83, p < .05), percent time in zone (t = -3.18, p < .001), fixation counts (t = -2.49, p < .05), percentage of total fixations (t = -2.27, p < .05), total fixation duration (t = -2.51, p < .05), and gaze-point counts (t = -2.48, p < .05) for the participants demonstrating high performance. All eye-movement data were higher in the pilots demonstrating high performance than in those demonstrating poor performance. However, the eye-movement data of pilots with different flight experiences did not reach a significant difference, which was an unexpected result.

Previous studies have shown that flight experience was significantly correlated with flight performance. Thus, we asserted that the effect of flight experience cannot be underestimated. The analysis result showed that the interaction between flight performance and experience achieved a significant difference in fixation duration (F = 4.89, p < .05), indicating that affected fixation duration. This result is consistent with that of most previous studies (as shown in Table 1).

Table 1. Summary Table of Two-Way ANOVA of Flight Performance and Experience
Full size table

Pilot Attention Allocation in Air-to-Surface Tasks. The trajectory of eye movement is spatial measurement. After the total of the movement distance of all fixation points were derived, a long distance represented an inefficient scanpath [18]. We analyzed the key stages that required the most attention in the task, which was the 10 s from flatten to off. The result indicated that the pilots demonstrating high performance exhibited a significantly shorter glance distance compared with the pilots indicating low performance (t = 2.46, p < .05). However, no significant difference was reached between pilots with different flight experiences, as shown in Table 2.

Table 2. Summary table of statistical analysis results in scanpath
Full size table

Pupil Size Variation of the Pilots in Executing Air-to-Surface Tasks. The pupil size variation can be regarded as a mental workload indicator. The analysis results of pupil size variation at the key stage showed that the average pupil size of the pilots at the flatten stage was approximately 24,634.87 pixels, which was expanded to 26,928.69 pixels during the tracking stage and constricted to 22,841.32 pixels during the off stage. Subsequently, a repeated measures design was used to analyze pupil size. The result revealed that the pupil size variation exhibited a significant difference (F = 20.17, p < .05). A post hoc analysis result showed that the pupil size at the flatten and pick-off stages was significantly larger than that at the off stage, as shown in Table 3. In addition, we test the mental workloads of difference in flight performance. The results indicated that the pupil size variation did not reach the level of significance at the flatten, tracking, and off stages.

Table 3. Summary table of the analysis results of pupil size variation at key stages
Full size table

4 Discussion

Effect of Flight Performance and Experience on Eye-Movement Performance. When pilots execute air-to-surface tasks, they must be highly focused on the fitness of external environments and messages from internal devices. If pilots cannot control the timing to complete bomb release while hitting the target accurately within a limited time, the flight task is considered a failure. The results showed that, the pilots primarily placed the fixation on the HUD. The pilots with different flight performance exhibited differences in eye movement. The fixations demonstrate the attention distribution and scanpath models of pilots. When pilots enter the key stage, they must rapidly grasp the information and target parameters to complete preparations. Thus, the pilots demonstrating high performance tended to exhibit increased fixation points, a longer fixation duration, relatively concentrated fixation distribution, and stable scanpath compared with the pilots demonstrating poor performance. Conversely, the pilots showed poor performance because of dispersed fixation points and long glance distance. Unable to obtain required information stably, pilots demonstrating poor performance may be distracted by an increasing amount of and diverse information processing, which increases mental workload and affects flight performance.

Cognitive Processing Processes of the Pilots. According to the experimental data and recorded videos, we determined that the pilots first focused their attention outside the cockpit to visually identify the target before bomb release. After the pilots identified and locked onto the target, they modified the relative position of the target to under the bomb release line of the HUD to facilitate aiming. In addition, because the pilots had to perform tactical lifting to escape safely after bomb release, the pilots changed the fixation to the position displaying the G force and adjusted G-force values as regulated to successfully complete the escape action (Fig. 1).

Fig. 1.
figure 1

Target tracking process (F1: initial assumed fixation; F2: location of target region; F3: bomb release aiming point; F4: fixation after bomb released).

Full size image

In the scanpath model, the pilot used a top-down cognitive processing model to search for the target during the bomb release task. When the target was in sight, the pilot was attracted by the stimulus and locked onto the target. Subsequently, flight knowledge and experience was used to modify the target to the optimal aiming location to facilitate precise tracking and bomb release. Overall, the cognitive processes at this stage comprised composite top-down and bottom-up cognitive processes, indicating a mutual cognitive processing process.

In practice, during the entire process of executing flight tasks, the top-down and bottom-up cognitive models appear constantly and interactively. The flight knowledge and techniques of pilots can be accumulated through training and experience. The awareness to stimuli and visual kinesthetic ability of pilots are related to their talent. However, visual search and attention allocation can be improved through training. Thus, in addition to flying technique training, training in visual ability and attention allocation must be equally emphasized to improve the overall flying ability of pilots, thereby improving their combat quality and flight safety.

Pilot Attention Allocation and Mental Workload. Hoffman proposed that places allocated with increased time and fixations are places receiving concentrated attention [13]. In the tracking process of air-to-surface tasks, pilots must focus on the information displayed by the HUD. The action must be completed in 5 s from the flatten stage to completing bomb release. After bomb release, the off stage must create 5–5.5 G forces within 2 s. This stage comprises the highest workload for pilots as a highly dynamic flight task. In addition, when tracking the target, pilots must match the bomb release line and the target in the HUD to hit the set target accurately and complete the task successfully. Air-to-surface tasks include numerous safety factors and concerns, the pilots must maintain an optimal physiological and psychological status to attain superior performance.

The study results revealed that the pilots exhibited similar variations in pupil sizes. When executing the flatten action, the pilots paid attention to the posture, altitude, and flatten angle of the aircraft, which increased the mental workload. Subsequently, the pilots immediately prepared for target tracking and bomb release, which did not reduce mental workload; instead, the pupil size expanded slightly. After completing bomb release, the pilots needed only to lift the aircraft off the field, which rapidly reduced the pupil size to the normal size. The result clearly indicated that the mental workload at the off stage was lower than that at the previous two stages. In addition, van Orden, and Makeig indicated that when the workload of air traffic controllers increased, their pupil diameter increased [19]. Kahneman and Beatty conducted a memory task study and observed that when the memory task exhibited high difficulty, the workload and pupil diameter increased [20]. Thus, using pupil size variation to predict workload is supported by the literature. Moreover, pupil size variation can be used to examine the mental workload of the pilots.

5 Conclusion

According to the research results, we propose several practical suggestions for applications. First, in addition to flying technique training, the attention allocation and cognitive abilities of pilots should be included in training courses. Focus is a key to increasing flight performance. How to focus on the appropriate and correct position must be trained and reminded. The region people focus on is where they pay attention. Accurate and efficient scanpath and gaze can improve the operating performance of pilots. Second, we suggest adding an eye-tracking system in simulation platforms to facilitate monitoring. Personal visual search tracks are affected by the surrounding environment and currently operating tasks. Thus, knowledge accumulated through learning leads to visual search behavior, which is crucial and practical for pilots in operating aircrafts [21]. The eye-tracking system can clearly record the eye-movement states of pilots and provide feedback to pilots and improving search models, thereby elevating the training effectiveness.

The most essential empirical contribution of this study is the in-depth discussion on the effect that flight performance and experience have on eye-movement performance. In addition, studies applying eye-movement techniques to improve flight training and attention management have received considerable attention in aviation safety fields. The result of this study confirms the effect of flight experience and performance on eye-movement performance. Moreover, we propose that the cognitive processing model of pilots during flight tasks can be used as empirical data references for training units to improve the practical precision of training. However, although we preliminarily analyzed and discussed pupil size variation and demonstrated pupil size variation during mental workload formation, adding physiological indicators to clarify the effect of cognitive and mental workload on pilots can further benefit aviation safety and operating performance.