Keywords

1 Introduction

1.1 New Technologies for New Cockpits

The safe operation of a modern aircraft relies largely on pilots interacting with complex visual displays, through which much of their flight information is presented. These aircraft use primary flight display (PFD) or head-up display (HUD) technologies to present this critical information such as aircraft airspeed, heading, course, and altitude. However, a revolutionary new technology entering the modern flight deck is the development of Synthetic Vision Systems (SVS), which has the capability to merge a computer-generated image of the outside world under that of the pilot’s traditional primary flight instruments. At its heart, SVS is driven by an elaborate database of topographical and cultural information that gives pilots a 2D synthetic image of the outside world (i.e. visually the lay of the land). By leveraging the power of Global Positioning Systems (GPS) and inertial reference systems, SVS displays depict the outside world in real-time, as an aircraft makes its journey from departure to destination (Fig. 1).

Fig. 1.
figure 1

The Collins Proline Fusion SVS PFD (and SVS moving-map to the right)

Full size image

Advances in GPS and operational protocols for lower minima approaches have created a positive climate for SVS. Having the added element of SVS data superimposed onto the PFD embodies the cues that are available to pilots in clear weather conditions. Research shows that modern electronic flight displays (particularly those incorporating SVS) have led to more efficient instrument scanning, more informed aeronautical decision making, and improvements in situation awareness (SA) [1,2,3].

An early demonstration of information visualisation supporting improved SA was performed by taking tabular data of thunderstorms and transforming it into 3D images. Users were able to understand much more about the forthcoming storms by looking at a few seconds of graphical information rather than spending much longer looking at a variety of data in textual formats [4]. By combining disparate data sources into a unified graphical presentation, findings such as these suggest that there is potential for SVS to support pilots in maintaining aircraft state awareness and ‘outside world’ awareness with markedly less effort.

Another application of this technological advance is in relation to how GPS is revolutionising landing procedures into airports surrounded by significant terrain, allowing pilots to fly curved approaches, rather than conventional linear (i.e. straight-in) approaches. ‘Pathway in the sky displays’ are the recommended method for safely flying these complex approaches [5] yet these approaches will eventually become contingent on having SVS. Whilst there is currently no operational advantage to be gained from SVS - such as allowing crew to fly to lower minima - it is a response to many safety issues characterised by real-world events. Thus, another prospect is that by using information visualisation, SVS might help reduce one of the most pervasive causes of aviation accidents to date: Controlled Flight into Terrain (CFIT) [6, 7].

1.2 Implications for Cognitive Performance

Flight displays design should provide pilots with the tools to support state change detection and maintain SA. Bringing another tool into the flight deck needs to be done in sympathy with already well-established operational protocols and work patterns otherwise it might compromise our natural cognitive capabilities. It is important to understand new technologies from a systems perspective, “new functionality and new technology cannot simply be layered onto previous design concepts because the current system complexities are already too high. Better human-machine interfaces require a fundamentally new approach.” [7, p. 7].

Research using a combination of human and performance modelling shows SVS can be introduced with minimal impact to current pilot scan patterns. Trials revealed that pilot dwell time on the PFD and Navigation Display (ND) deteriorated when SVS was presented on a separate screen. Whereas, performance-modelling trials alone determined that combining the PFD with SVS provided the best configuration, requiring fewer glances away from the primary flight instruments [8]. Nevertheless, one must approach these results with caution, since findings were based on a simplified representation of human performance as demonstrated by a computer, which may not wholly represent the real-world dynamics of human behaviour.

Empirical data from witness testimony, computer gaming accuracy and automobile safety, show that even changes within the direct field of vision can be misinterpreted or even missed entirely (a phenomenon known as ‘change blindness’) when presented with too much surrounding detail [9, 10]. Research has also shown that people have great difficulty simultaneously attending to two visually superimposed scenes [11,12,13]. Similarly, simply increasing background colour, hues and depth has been found to impair visual processing [14]. Visual perception has been proven to be quite vulnerable to these effects, yet in the aviation domain, little research has been conducted into the operational practicality of SVS and aspects of change-blindness.

Change blindness refers to the difficulty people have in readily detecting changes outside the very small region of focussed attention [15, 16]. A fundamental principle of change detection is that it is a daily problem that people simply cannot attend to all the objects around them [17]. An object must be attended to, to be seen to change [17, 18]. The ability to see in high definition is restricted to a very small area around the focal point of the eye, and so perceiving detail in the environment is incumbent on effective and frequent eye movements [19]. In a simple scene, this does not take much effort, but increasing the amount of information vying for one’s attention will place increased cognitive demands on the observer, thus slowing the visual search or increasing opportunities for them to miss information or state changes in the visual scene.

Previous research has demonstrated that there is greater likelihood of a failure to integrate information across saccades, distractions or ‘mud splashes’ or ‘Flicker’ episodes (i.e. techniques used to investigate change blindness in laboratory conditions) compared with changes introduced during direct fixation [19, 20]. Furthermore, how meaningful information might be is a key factor in determining what people decide to process, and thus becomes an important factor in change blindness [19]. As a whole, what these findings illustrate is that just because one’s eyes are open, does not necessarily mean everything is being seen.

A conclusion from the National Weather Service (NWS) best illustrates the argument for SVS - the use of numbers requires analysis, but the use of imagery induces intuition [4]. However, although enough evidence exists for the merging of SVS with the PFD, decades of research challenges the practicality of this system. Naturalistic scenes interfere most with target searches [10, 21] and changes are masked by the sheer amount of colour or detail on a display [22].

Essentially, having a highly accurate depiction of the outside world could be considered intuitive, but increasing the colour and dynamics within these displays could also have an adverse impact on a pilot’s ability to notice changes or transitions, and sustain attention to other key visual areas.

1.3 Rationale

A virtual SVS depiction of the external environment may be intuitive and improve situation awareness by helping pilots to scan their displays more efficiently. However, despite this operational benefit, relatively little research has explored the relationship between sophisticated SVS displays and pilot performance.

Building on previous research into change blindness, the aim of this study was to investigate any response time difference for pilots detecting visual changes occurring on a conventional flight display compared with an SVS superimposed PFD. A further element of the research explored the topic of change detection performance as a function of expertise. It was predicted that changes occurring within conventional PFDs would be detected quicker than changes occurring within SVS PFDs (since increasing detail, motion and colour of backgrounds have been shown to hamper visual detection speed of cues) and due to more developed scanning techniques and increased knowledge associated with greater flight time, captains would be quicker at detecting changes across both conditions than first officers.

2 Method

Using custom designed stimuli that simulated both conventional (non-SVS) and SVS flight displays, an experiment conducted to measure pilot response time to visual cues (as a surrogate measure for SA).

2.1 Participants

An opportunistic sample of 18 pilots consisting of nine captains (age 34 to 52 years) and nine first officers (age 23 to 34 years) who were qualified in both traditional as well as SVS avionics. Five captains and one first officer had over 5,000 h flying time: four captains and five first officers had 1,500 h to 5,000 h flying time; three first officers had less than 1,500 h flying time.

2.2 Apparatus

The experiment was presented on a Macintosh Apple MacBook Air 13-inch laptop. Some participants conducted the experiment remotely on their own laptop computer. A ‘Flicker Paradigm’ was developed using Inquisit Software with the custom-made images. Participants generally positioned themselves as close as possible to the seating position they took in the cockpit. General flight deck anthropometrics have pilots seated anywhere between 500 mm to 700 mm from the display, with the screen in the centre of the visual field (within 30° of binocular view) and approximately 15° below the normal line of sight [23].

2.3 Design

The study employed a 2 × 2 mixed design. The between groups factor was ‘experience’ (captain vs first officer) and the within-groups factor was ‘display’ (conventional vs SVS). Dependent variable measures were collected for response time data to on-screen changes in the flight instrumentation displays.

A counter-balanced repeated measures approach ensured that the order of the stimuli was randomised for each participant to remove any learning, practise or fatigue effects.

The Flicker Paradigm is an adaption of tests that simply present an original and modified image back to back. By inserting a blank screen between each image (i.e. a ‘flicker’), fixated attention would be required to notice any changes and coincident with a natural blink since pilots do not typically dwell on the PFD for more than 30–40 s at a time [24], or a simultaneous on-screen change to an icon [25].

Images simulating flight displays were designed in accordance with the AC25-11A/B and DEF- STAN standard for PFDs, and presented to the participants in pairs (original and modified). Each image pair was identical, except for one single change (Figs. 2 and 3). A ‘change’ was defined by; an object disappearing or reappearing, an object changing colour, an object changing position, or an alphanumerical value change. These changes were only incorporated into the flight symbology that a pilot would normally be expecting to change (i.e. a colour of a speed read-out was not changed to a colour that would be inconsistent with reality). Although the changes are highlighted for ease of identification in Figs. 2 and 3, they were not in the experiment and so participants had to judge for themselves what change might have occurred.

Fig. 2.
figure 2

Ground Speed reading change between left and right displays (highlighted on right hand display in bottom left panel for ‘GS’)

Full size image
Fig. 3.
figure 3

Localiser displacement indication change between left and right displays (highlighted on right hand display under centre white arrow)

Full size image

The original image was repeatedly alternated with a modified image (240 ms each), separated with a Inter-Stimulus Interval (ISI) (a blank screen) lasting 80 ms. The images and ISI’s were alternated until the participant detected the change or 60 s had passed [26], whichever came first. The sequence was looped as ‘original’ > ‘original’ > ‘ISI’ > ‘modified’ > ‘modified’ > ‘ISI’. This created a degree of uncertainty about when stimulus change might occur [18] and also gave participants more of an opportunity to process each image. This made the search more naturalistic and participants were less able to predict an oncoming change.

2.4 Procedure

Participants were shown a total of twenty pairs of images presented across the two conditions: 10 pairs of conventional PFDs without SVS and 10 pairs of PFDs with simulated SVS information. Each pair was identical, except for one change. Participants were tasked with detecting the change. Pushing the spacebar when the change was detected automatically logged their response time while also triggering the next image pair in the sequence. Immediately after each image pair the participant was asked to report the type of change they saw by typing their observations into a dedicated text box. This was done to help ensure accuracy and prevent against random clicking.

3 Results

Data from 18 participants were included in the final analyses. Within this sample 50% were captains (N = 9) and 50% were first officers (N = 9). Analysis of Variance tests were conducted on the response time data.

3.1 Response Time as a Function of Display Type

A significant main effect for ‘display’ was observed (F = 4.9/Fcrit = 4.15, p = 0.034) illustrating that display complexity influenced response time. Pilots were slower at detecting changes occurring on the SVS display (M = 25 s, SD = 9 s) than Conventional displays (M = 19.5 s, SD = 7 s). From the sample, 83% of pilots were faster at detecting changes on the Conventional display, whereas just three participants were quicker in the SVS condition. For changes detected in the first 40 s pilots noticed changes in 89% of the trials for the conventional display compared with only 78% for the SVS condition.

3.2 Response Time as a Function of Pilot Expertise

A significant main effect for ‘experience’ was observed (F = 6.13/Fcrit = 4.15, p = 0.019), illustrating that experience affected the response times for changes on the displays. However, this was in a different direction than anticipated. Less experienced first officers were on average 6 s faster, and more accurate (i.e. they noticed more changes in the displays) than the more experienced captains. Captains missed 17% of the changes presented to them while first officers missed only 8% of the same changes. For changes detected within the first 40 s, captains noticed changes in 78% of the trials and first officers noticed changes in 89% of the trials. Overall co-pilots demonstrated quicker change detection than captains on both display types. This was reflected in results for an interaction (display x experience) although this was not statistically significant (p = 0.8).

4 Discussion

4.1 Response Time as a Function of Pilot Expertise

The goal of SVS is to help pilots make more informed decisions by providing them with a clearer picture of their surroundings. The purpose of this study was to explore if these intricate displays increased the pilots’ information processing burden. Specifically, if the increased detail in the displays hampered their ability to scan the flight instruments effectively and notice critical changes in the information before them.

Findings from prior research has collectively found that people tend to become distracted and overburdened by increased background detail [22]. This study hypothesised that the complexity within the flight display would negatively impact on change detection speed. This appeared to be supported by the results as significantly slower detection times were observed to changes occurring on the SVS displays.

Although the results from this experiment do not refute that imagery makes for a more efficient and intuitive scan, they do indicate that small on-screen changes can be masked. Yet, while the primary goal of this research was to explore some of the pitfalls of the new SVS display designs, it also brought to light some of the underlying human factors issues unique to corporate aircraft operators. The misguided universal assumption that pilot expertise can be indexed by age and hours, and how multiple users from an array of ages, generations and ranks are being expected to operate highly sophisticated, un-customisable technology.

An important cognitive skill for being an effective crewmember is the ability to scrutinise a myriad of information in a timely fashion. Yet an important finding was that captains were generally slower than first officers at spotting changes in both display formats (i.e. conventional and SVS). It may have been possible to surmise that careful, knowledge-driven searches are characteristic of the more senior ranking and experienced crewmembers, which might explain the longer detection times. However, one clue to understanding why the first officers excelled in the change detection tasks could be explained by the renovation of the fleet of training aircraft around the world. Over the last decade, virtually all new aircraft are manufactured with some degree of digital (glass) cockpit technology (Fig. 4, left-hand image). Many of the newer pilots may never have even flown with traditional analogue gauges as the more experienced captains may have done in the past (Fig. 4, right-hand image).

Fig. 4.
figure 4

Glass cockpit display (left) and traditional analogue display (right)

Full size image

4.2 New Pilots for New Cockpits

The new generation of training aircraft might, in part, be the reason for the generation of new first officers performing better on digital displays (i.e. conventional and SVS displays) than their captains. In this situation, expertise might be more attributable to circumstance and contextual learning (i.e. being exposed to technology throughout one’s early life and training) and less a function of hours and rank.

Taking that same logic, captains, with the foundation of their skills and proficiency honed using analogue and conventional displays, could find themselves coping with distractions from the additional information and learning unfamiliar displays. This view is consistent with research into skill acquisition, that when knowledge is first acquired it is organised in a way that, thereafter, can be accessed automatically through pattern-based retrieval [27, 28]. The cues available to captains on newer electronic displays may not complement their pre-existing patterns or mental models, thus requiring a period of readjustment or acclimatisation to better interact with SVS displays.

These findings resonate with the statement, “too often, when a new system is introduced, it is assumed that trainees are already experts in the processes the system is intended to monitor and control” [29, p. 444]. A pilot who is proficient and can scan their instruments in a largely automatic fashion, will not naturally be equally proficient with a highly advanced SVS display. Thus, it is conceivable that captains performed worse because they had to commit more attentional resources and expend more effort engaging in active learning, in order to overcome their pre-existing mental models. Perhaps this reveals a generational challenge that new computerized technology might not be able to be embraced by all users as a ‘one-size-fits-all’ solution. As with other sophisticated devices, such as mobile phones and computers [30] changing technologies on flight decks will need to be tailored to the needs, requirements and limitations of a wider selection of users.

While the findings of this research provide a tantalising insight into the way that new technologies may emerge in new applications, it is important to bear in mind that the sample population in this study was not representative of the pilot population at large. As such, the variance in performance between captains and first officers may need to be explored and verified with further research. To better simulate the tasks and workload that accompany the normal flight regimen, future improvements to this experiment can be achieved by carrying out SVS change blindness trials in simulator environments while also using a more industrywide representative crew sample.

Within commercial transport operations, for instance, generally the captain and the first officer will take control of alternate flights, getting roughly an equal number of take-offs and landings. Whereas, in charter aviation, the first officer may be restricted from acting as the ‘pilot flying’ until they have logged several hundred hours of flight time. Depending on proficiency, in some rare cases, first officers could spend the better part of a year operating only as ‘pilot monitoring’ logging only a handful of take-offs and landings. This implies that they may have developed an affinity for scanning displays, which may have contributed to their increased change detection performance. Finally, there may also have been a lack of engagement among the more senior crew members (i.e. captains) and perhaps an increased engagement among the junior crew members (i.e. first officers) who were more conscious of their performance.

5 Conclusion

Overall, these findings leave us with the view that complex and sophisticated systems can become a challenge, even for experienced users. By providing a graphical depiction of the outside world environment, SVS can augment pilot SA and aircraft state awareness beyond anything previously introduced; and be necessary for more advanced approaches into precipitous terrain. However, all too often with rapidly developing technologies, the potentially negative side-effects are excluded from initial exploratory studies. Two strongly counterintuitive results arise from the findings of this experiment, indicating that there is perhaps another side to the coin with SVS.

SVS presents a tantalising and unsurpassable method of providing situation awareness. Therefore, it cannot be reasonably placed on a separate display without causing significant decrements in visual scanning. However, given the potential risks associated with decrements in change detection, there may be a requirement for SVS to be assuaged during phases when distraction would be most critical.

Also brought to light was how expertise is not only characterised by hours, but by the ability to adapt to constantly evolving technologies. The first officers performed their tasks more quickly and more accurately than captains. This was unexpected considering that experts (based on hours and rank) were purported to conduct more effective scans. Maybe this denotes a failure to consider the way SVS equipment might interact with the already established behaviours and mental models built up through years of exposure on older technologies. Or perhaps pilots need to be more mindful of the way they interact when presented with these powerful new technologies.