1 Introduction

Pilots’ operation performance can affect flight safety directly. Many studies have reported that pilot error is the primary cause of over 60% of flight accidents [1, 2]. The statistics on commercial flight accidents in China from 2006 to 2015 indicated that flight crew factors contributed to 64.58% of accidents [3]. Particularly in the final approach and landing stage, the occurrence rate of pilot error is significantly higher than in other phases because pilots need to deal with more situational change, greater decision making, and greater operational activity [4,5,6]. Accident statistics have also indicated that approach and landing was the most dangerous phase of flight; the landing phase in particular accounted for 23% of total fatal accidents occurring from 2006 to 2015, despite the fact that it accounts for just 1% of average flight time [7].

In the field of landing safety, many previous studies have focused on pilot visual perception and pattern analysis [8,9,10,11,12], runway overrun risk modeling [6, 13], critical factor analysis [14,15,16], and so on. Wang [17,18,19] applied a new method for landing safety research by using real flight data to analyze performance features of long landing incidents. However, there was relatively less research of landing performance. In particular, there were lesser outcomes regarding with performance and operation analysis based on real flight data.

The flight quick access recorder (QAR) is a system that can acquire aircraft operational data easily. It includes airborne equipment for recording data and a ground software station for storing and analyzing data. The QAR can record all kinds of aircraft parameters, pilot operation parameters, environmental features, and alarm information during an entire flight. The practice has proved that QAR data are helpful for improving flight safety management and quality control. However, the data have been rarely utilized in research.

The main purpose of this study is to develop a flight landing operation performance evaluation system based on flight quick access recorder data and a risk evaluation model. The system is expected to be used to evaluate, analyze, and pre-alarm the performance of the landing operation of the pilot after the flight task, to provide practical technical support for airlines to monitor pilots’ landing operation performance and landing risk.

2 Methodology

2.1 Quick Access Recorder Data

QAR data can record the flight and operation of an airplane, information about its environment, and other types of information. The QAR data sampling frequency can reach as high as 16 Hz in modern aircraft. The Civil Aviation Administration of China (CAAC) has implemented the Flight Operations Quality Assurance (FOQA) program since 1997, with all commercial airplanes of Chinese airlines obliged to install a QAR or similar equipment [20]. Based on related operational rules and regulations, commercial airlines always use flight QAR data to monitor and analyze the entire aircraft and pilot operation performance in flight. When a flight parameter exceeds the prescriptive normal range, it is called a QAR Exceedance Event or Unsafe Event. Exceedance events usually do not lead to severe results, but they can increase the probability of an accident and bring potential harm to aircraft and even passengers.

Most flight operation departments in airlines utilize the flight data in a simple way. They generally just use the data to monitor flight safety status by counting the numbers of unsafe events. Obviously, this could not make full use of data resources through the simple logic of ‘exceedance management’. Especially when the flight parameter of unsafe event is close to but not exceeds the threshold value, a large amount of data would be wasted. Therefore, a more effective method of using flight data is expected to be developed from the user-centered perspective.

2.2 Evaluation Model

There are large differences in the actual landing process. Pilots often must implement some emergency operations due to special circumstances, different pilots have different flight operation habits, and different aircraft also have different performance characteristics, so it is difficult to evaluate a pilot’s operation performance directly. However, no matter the environmental factors, aircraft factors, and pilots’ individual differences, the goal of the landing is the same, which is that the aircraft land in standard position on the runway with the right pitch attitude, an appropriate speed, and an appropriate load. A pitch attitude exceeding the standard may result in tail strike incidents, a vertical load that is too heavy may result in a hard landing, and a pilot missing the standard landing point may cause an overrun runway accident. This means that pilots should try to reduce the risk of overrunning the runway, sustaining a hard landing, or causing tail striking when the aircraft touches down. Therefore, we introduce the concept of risk evaluation to evaluate the operation performance of pilots by calculating the risk of these 3 abnormal incidents after each landing. The risk value of these 3 landing incidents is taken as the index of landing operation performance evaluation.

Risk is a 2-dimensional concept that is usually measured based on 2 indicators. One is the severity of the consequences of an incident, and the other is the occurrence probability of an incident. The 3 parameters of touchdown distance, vertical acceleration, and pitch angle, which can be recorded by a QAR, are generally used as evaluation indexes for judging landing incidents. Theoretically, each kind of flight parameter distribution will be approximately a normal distribution in a period. If the distribution functions of the 3 parameters are obtained, it is possible to evaluate the severity of the consequences of landing incidents and the probability of these incidents occurring according to the distribution functions and the algorithms, and then the risk value of landing incidents can be calculated. Then, the final landing operation performance values can be calculated and combined with the weights of the 3 types of incidents. Based on the above analysis, the evaluation model of landing operation performance is written as follows:

$$ \left\{ {\begin{array}{*{20}l} {P_{landing} = \omega_{1} \cdot R_{TD} + \omega_{2} \cdot R_{VA} + \omega_{3} \cdot R_{PA} } \hfill \\ {\omega_{1} + \omega_{2} + \omega_{3} = 1} \hfill \\ \end{array} } \right. $$
(1)

\( P_{landing} \) is the value of landing operation performance; \( \omega_{1} \), \( \omega_{2} \), and \( \omega_{3} \) are the respective weights of the touchdown distance, vertical acceleration, and pitch angle, which can be determined from statistical data of landing incidents or the Delphi method, or determined and adjusted by the safety supervision department of an airline according to the safety situation and monitoring strategy. \( R_{TD} \), \( R_{VA} \), \( R_{PA} \) are respective risk values of the 3 landing incidents (runway overrunning, hard landing, and tail striking). They can be calculated by formula 2 below:

$$ \left\{ {\begin{array}{*{20}l} {R_{TD} = P_{TD} \times S_{TD} } \hfill \\ {R_{VA} = P_{VA} \times S_{VA} } \hfill \\ {R_{PA} = P_{PA} \times S_{PA} } \hfill \\ \end{array} } \right. . $$
(2)

In formula 2, \( P_{TD} \), \( P_{VA} \), and \( P_{PA} \) represent the probability of the 3 landing incidents of runway overrunning, hard landing, and tail striking, respectively, and \( S_{TD} \), \( S_{VA} \), and \( S_{PA} \) represent the severity of the consequences of these respective landing incidents. These six parameters could be calculated out from the flight data distribution function of touchdown distance, vertical acceleration, and pitch angle. Then we have the value of landing operation performance combined with the weights of the 3 incidents, and the level of landing operation performance can be calculated out. The algorithm has been mentioned in our previous study [21].

3 System Design

In the last section, the landing operation performance evaluation model was established based on flight QAR data and risk evaluation theory. Three landing operation performance evaluation indexes (touchdown distance, vertical acceleration, and pitch angle) of each flight could be used to objectively evaluate the flight landing operation performance according to the model and algorithm. In this section, the flight landing operation performance evaluation system (FLOPES) will be introduced.

3.1 System Hierarchy

The flight landing performance evaluation system was designed to including 7 modules: flight data processing, operational performance evaluation, unsafe event inquiry, and statistics, flight operation instructions, user center, user guidance, and system administration. The hierarchical structure of the system and each sub-function module of the system are shown in Fig. 1.

Fig. 1.
figure 1

Hierarchy diagram of flight landing performance evaluation system

Full size image

3.2 System Logic

The logic diagram of the flight landing performance evaluation system is shown in Fig. 2.

Fig. 2.
figure 2

Flow diagram of flight landing performance evaluation system

Full size image

4 System Development

4.1 Development Environment and Process

Based on system design and database design, the MyEclipse tool is used to develop the flight landing operation performance evaluation system on the J2EE platform. Meanwhile, the SQL relational database management system is used to reduce network bottlenecks and improve data transmission efficiency.

4.2 System Interface and Functions

The developed Flight Landing Operation Performance Evaluation System (FLOPES) includes 7 modules, such as flight data processing, operational performance evaluation, and unsafe event inquiry and statistics. The main interface is shown in Fig. 3. The main interface includes a menu bar and links to 6 functional modules.

Fig. 3.
figure 3

Main interface of FLOPES

Full size image

The core function of FLOPES is to evaluate landing operation performance using flight data. The entire evaluation algorithm of this function is illustrated in the last section. After clicking the “Operation Performance Evaluation” icon in the main interface and finishing the model parameters setting, the system will enter the calculation page. When the calculation is completed, the system will provide a prompt box and jump to the evaluation result page, as shown in Fig. 4. The evaluation results can be exported as Excel files for further use.

Fig. 4.
figure 4

Evaluation results of flight landing operation performance

Full size image

Another important function of FLOPES is to provide users with unsafe event inquiries and statistics. Users can enter the performance evaluation inquiry page and the event inquiry statistics page by clicking “Unsafe Event Statistics” on the main interface. After inputting the information regarding the captain, flight date, flight number, and aircraft type, the system will indicate the relevant evaluation records and statistical results, as shown in Fig. 5.

Fig. 5.
figure 5

Operational performance evaluation result inquiry and statistics

Full size image

Meanwhile, the system supports importation of QAR data into the system database and carrying out the corresponding inquiry. The user can inquire about relevant event information based on the entered information of captain, flight date, flight number, aircraft type, and event type or flight phase.

5 Conclusions

The Flight Landing Operation Performance Evaluation System was introduced in this study. The system was tested in the flight quality control department of an airline, and the QAR data of the flight was imported in batches to carry out flight data processing, landing operations performance evaluation, and other operations. The trial results showed the following:

  1. (1)

    The system can accomplish all basic functions, from the input of basic information and parameters to the output of evaluation results. It achieved landing operation performance evaluation, flight data processing, event inquiry and statistics, flight operation guidance management, user management, and other functions, indicating that the integrity of the system is good.

  2. (2)

    The system provides a support tool for flight operations quality assurance (FOQA) and flight training. The system can evaluate the performance of the landing operation of a flight that is more objective, effective, and reasonable than the simple overrun event management in current flight quality monitoring. It can provide a more accurate and objective basis for airline performance rewards and punishments.

  3. (3)

    The system provides actual data support for the flight operations department to monitor and control the landing risk. However, the system needs to be improved for shortening its response time when there is a mass data inputting and processing.