Keywords

1 Introduction

1.1 The Status and the Meaning of Research About Comprehensive Evaluation on Cockpit Ergonomics Design

According to literature and accident report, about 70 % of all aviation accidents were caused by human factors [1–4]. In order to reduce human error and pilot workload, the flight deck need to be designed carefully with full consideration of ergonomics. Then the aviation safety can be enhanced. In order to ensure the design quality of man-machine interface on flight deck, the ergonomic requirements need to be integrated in design and the ergonomic evaluation efforts are essential. However, it is difficult to grasp the overall ergonomic design level of cockpit. There is a need for comprehensive evaluation. In the evaluation and comprehensive evaluation aspect of cockpit ergonomic design, plenty of research and application efforts have been done [5–8].

1.2 Limitations of Existing CE Methods

The process consisted of 5 steps, i.e. specify the evaluation purpose, identify the cockpit ergonomic metrics (evaluation indicator system), determine indicators’ weights; rate each evaluation indicator, and select/construct comprehensive evaluation model to conduct comprehensive evaluation [9, 10].

Weight determination is a key step in the process of comprehensive evaluation. The rationality of the weight represents a correct description of the relationship of evaluation metrics and object to be evaluated, which determines the validity of the result of CE, apparently. Several methods have been proposed to determine weights Majority of them can be classified into subjective methods, objective methods and combinations of them depending on the information provided [11]. In general, objective methods assign indicator weights according to the structure or the internal mechanism of evaluation object. However, for the comprehensive evaluation problem of cockpit ergonomic design, the relationship between various indicators is complex, the effect of each individual indicator on the overall ergonomic level is not very clear. Moreover, ergonomic evaluation is greatly influenced by evaluator’s subjective factors. These reasons make it difficult to determine indicator weights using objective methods. Thus, in general case, the subjective method, which is based on expert judgment, is commonly used to acquire indicator weights. However, the subjective weighting method has the following limitations:

  • Strong subjective, the weighting results depend on experts’ work experience, knowledge structure and their preference,

  • Expert judgment is equivalent to the black box operation and the transparency of the evaluation process is poor,

  • And, since the uncertainty of subjective judgment, the repeatability of results is poor.

Therefore, the same decision maker can give different weight to the same indicators under different situations. In order to achieve group consensus, Mukherjee (2014) advised the Brian Storming method [12], and Dempster–Shafer theory was used in Ju and Wang’s (2012) research [13].

Comprehensive evaluation is a method using a mathematical model \( (y = f(\varvec{w},\varvec{x})) \) to aggregate individual indicator evaluation result into a composite indicator with consideration of the weight information. The comprehensive evaluation value represents the overall level of the evaluated object. Here, the mathematical model used for “aggregation” is the comprehensive evaluation model. Due to the entire cockpitis a complex system, including many subsystems, and there exist closely correlation and strong coupling between the subsystems. This makes the evaluation indicators for the comprehensive evaluation of cockpit ergonomics have significant interdependence. Whereas the existing comprehensive evaluation models, such as fuzzy comprehensive evaluation method, principal component analysis method, andutility theory model, etc., are essentially a linear weighted method. Their results contain serious duplicated information and cannot reflect the true level of the object evaluated [14]. Although the neural network is a non-linear model, it is difficult to obtain enough high quality training samples, which greatly limits its application.

1.3 Innovation Work

The purpose of this paper is to address the limitations described above. With the feedback of the majority’s opinion of experts, Delphi method can achieve group consensus through multiple rounds of consultation [15]. Inspired by Delphi method, the feedback mechanism was introduced in Order Relation Analysis (ORA) method to address the bias and uncertainty of expert subjective judgment.

The existing practice to handle the interdependence between evaluation indicators is to eliminate the overlapping information during the construction stage of the evaluation metrics [16], or to adjust the indicator weight according to the correlation between them [17], when ANP is advisable [18]. Considering the general process of comprehensive evaluation, it is apparent that another potential point to handle the redundant information between indicators lies incomprehensive evaluation model. However the related literatures are few. The improved radar chart proposed in this paper has integrated the correlation between indicators, which is a means to address the overlapping information through the development of an appropriate comprehensive evaluation model.

1.4 The Structure of the Thesis

The remainder of this paper is structured as follows. The second section introduces the process of Delphi-ORA method in detail, including the design of Loop control variable and feedback variable. The third section describes the improved Radar Chart. Finally, a conclusion and discussion is given.

2 Delphi-Order Relation Analysis Method

The order relation analysis (ORA) method proposed by Guo [19], also called G1 method, is a subjective weighting approach based on expert judgment. Compared with analytic hierarchy process (AHP), ORA method does not need consistency testing. Besides, the amount of Comparative judgment work has been reduced greatly. For these reasons, ORA method was chosen in this paper for improvement. A feedback mechanism was introduced in ORA method, which was inspired by Delphi method. The Delphi-ORA method requires organizers to do statistical processing on the data collected in previous advisory round and feedback the result to experts. Then experts are required to give their opinion again based on the feedback information. After several rounds of consultation, experts’ opinion will eventually tend to be consistent.

2.1 ORA Method Process

ORA method firstly asks experts to sort the evaluation indicators according to their importance on evaluation object in a descending order. And then estimated values for the importance ratios between adjacent indicators should been given. The estimated ratios are always presented as qualitative linguistic values. During the subsequent data processing, a numerical value will be assigned to each linguistic value. Then through a simple mathematical treatment, the weight coefficient of each indicator can be calculated. The weighting process of ORA method consists of the following three steps.

Indicator Importance Sorting.

For given evaluation object, if indicator \( u_{i} \) is more (not less) important than indicator \( u_{j} \), then it can be documented as \( u_{i} > u_{j} \). First, the expert need to choose the most important indicator he think among then indicators \( \left\{ {u_{1} ,u_{2} , \ldots ,u_{n} } \right\} \), and record it as \( {u_{1}}^{{^{*}}} \). Then select the most important indicator from the remaining n−1 indicators to be as \( {u_{2}}^{*} \). Repeat the steps above, after n−1 rounds, a descending sorting according to their importance of the n indicators can be derived:

$$ {u_{1}}^{*} > {u_{n2}}^{*} > \cdots > {u_{n}}^{*} $$
(1)

Where \( {u_{i}}^{*} \) is the i-th indicator after sorting (i = 1,2,…,n).

Estimate the Importance Ratio.

Suppose \( {w_{i}}^{*} \) is the weight coefficient of indicator \( {u_{i}}^{*} \). Then the importance ratio (\( r_{i} \)) between indicator \( {u_{i - 1}}^{*} \) and \( {u_{i}}^{*} \) can be represented as \( {w_{i - 1}}^{*} /{w_{i}}^{*} \) (formula 2). Through expert judgment, a qualitative linguistic value of \( r_{i} \) can be obtained. Then a numerical value should be assigned to each linguistic level as appropriately. The assignment can refer to Table 1.

Table 1. Suggested linguistic levels and corresponding numerical values for \( r_{i} \)
Full size table
$$ {w_{i - 1}}^{*} /{w_{i}}^{*} = r_{i} ,\,\,\,\,\,\,\,i = n,n - 1, \ldots ,2 $$
(2)

Calculate Weight Coefficient \( w_{i} \).

Formula (3) and (4) can be used to calculate the weight coefficient of each indicator in sorting (1)

$$ {w_{n}}^{*} = \,\left( {1 + \sum\nolimits_{k = 2}^{n} {\prod\nolimits_{i = k}^{n} {r_{i} } } } \right)^{ - 1} $$
(3)
$$ {w_{i - 1}}^{*} = r_{i} {w_{i}}^{*} ,\,\,\,\,i = n,n - 1, \ldots ,2 $$
(4)

Therefore, weight coefficients of sorted indicators can be obtained,

$$ {\varvec{W}}^{*} = ({w_{1}}^{*} ,\,{w_{2}}^{*} , \ldots ,{w_{n}}^{*} ) $$
(5)

Then adjust the order of each element in weight vector \( {\varvec{W}}^{*} \), we can get the weight vector corresponding to the original indicator set \( \{ u_{1} ,u_{2} , \ldots ,u_{n} \} \),

$$ \varvec{W} = (w_{1} ,\,w_{2} , \ldots ,w_{n} ) $$
(6)

2.2 Information Feedback Variable

The consultation result of former round is provided to experts through information feedback variable (IFV). Assuming there are m experts participating in the indicator weight consultation of ORA method, then after the first round consultation, we can get m pairs of different sorting and corresponding estimated importance ratios (see formula 7) for indicator set \( \{ u_{1} ,u_{2} , \ldots ,u_{n} \} \).

$$ \left\{ {\begin{array}{*{20}c} {{u_{i1}}^{*} > {u_{i2}}^{*} > \ldots > {u_{in}}^{*} } \\ {(r_{i2} ,r_{i3} , \ldots ,r_{in} )} \\ \end{array} } \right.,\,\,\,\,\,(i = 1,2, \ldots ,m) $$
(7)

Thus, the weighting result for indicator set \( \{ u_{1} ,u_{2} , \ldots ,u_{n} \} \) of the i-th expert can be calculated using ORA method.

$$ \varvec{W}_{i} = (w_{i1} ,\,w_{i2} , \ldots ,w_{in} ),\,\,\,(i = 1,2, \ldots ,m) $$
(8)

Where, \( w_{ij} \) denotes the weight coefficient assigned to the j-th indicator in \( \{ u_{1} ,u_{2} , \ldots ,u_{n} \} \) by the i-th expert.

The arithmetic mean of \( \varvec{W}_{i} \) can be calculated as formula (9), which is regarded as the ultimate weight vector of the first round consultation.

$$ \varvec{W} = \frac{1}{m}\sum\nolimits_{i = 1}^{m} {\varvec{W}_{i} } = \left( {\frac{1}{m}\sum\nolimits_{i = 1}^{m} {w_{i1} } , \ldots ,\frac{1}{m}\sum\nolimits_{i = 1}^{m} {w_{in} } } \right) = (w_{1} ,w_{2} , \ldots ,w_{n} ) $$
(9)

According to the size of each element (indicator weight) in the weight vector W, the first round average sorting of the n indicators could be determined. And this average sorting is selected as the information feedback variable.

$$ {u_{1}}^{*} > {u_{2}}^{*} > \ldots > {u_{n}}^{*} $$
(10)

2.3 Loop Control Variable

Loop control variable is used to test the consistency of expert judgment, upon which the weight consultation process is determined whether to be ended. In this paper, an average ordering deviation index (AODI) based on the number of reverse order is proposed to represent the consistency/divergence level of expert group ordering.

An ordered array composed of natural numbers (1,2,…,n) is called an-order permutation, denoted as \( j_{1} ,j_{2} , \ldots ,j_{n} \). An n-order permutation can most have n! different permutations. Among them, the ascending permutation, i.e. 1,2,…,n, is defined as standard permutation or natural permutation. In a permutation, if one bigger number is in front of a smaller one, then these two numbers form a reverse order. The total number of reverse orders in a permutation is defined as the reverse order number (RON) of the permutation, denoted as \( \tau (j_{1} ,j_{2} , \ldots ,j_{n} ) \).

The average sorting \( ({u_{1}}^{*} > {u_{2}}^{*} > \ldots > {u_{n}}^{*} ) \) obtained from a weight consultation round is regarded as the standard permutation (1,2,…,n). For the i-th expert’s sorting \( {u_{i1}}^{*} > {u_{i2}}^{*} > \ldots > {u_{in}}^{*} \), it corresponds to a n-order permutation and its RON can be denoted as \( \tau (i) \). For a n-order permutation, the maximum RON is n(n−1)/2. The ratio of average RON of m permutations provided by expert group and the maximum RON is used to represent the consistency/divergence level of expert group ordering, i.e. the average ordering deviation index (AODI).

$$ {\text{AODI}}\,{ = }\,\frac{{\sum\nolimits_{i = 1}^{m} {\tau (i)/m} }}{n(n - 1)/2} $$
(11)

The smaller the ordering deviation is, the smaller AODI will be, and vice versa. If AODI < 0.1, the consistency of expert group ordering is acceptable and the weight consultation can be ended.

3 Improved Radar Chart

3.1 Radar Chart and Comprehensive Evaluation

Radar chart is also known as spider diagram, which consists of several concentric circles (or polygons) and some axes starting from the circle center. Each axis in the radar chart denotes an indicator, while each concentric circle (or polygon) represents a certain indicator level. Radar chart comprehensive evaluation model is a kind of indicator value aggregation method based on the extraction of feature variables of radar chart. Radar chart comprehensive evaluation model is a combination of graphical evaluation method and digital evaluation method, which is greatly suitable for an entire and overall evaluation of complex multi-attribute structure, and much more intuitive, as well (Fig. 1).

Fig. 1.
figure 1

The general form of traditional radar chart

Full size image

Huili Zheng described the general process of radar chart comprehensive evaluation [20], Liu and Chen Yong improved radar chart comprehensive evaluation respectively [21, 22]. The most commonly used feature variables in radar chart comprehensive evaluation are the area A and perimeter C of radar chart, which are calculated as:

$$ \left\{ {\begin{array}{*{20}c} {A = \sum\nolimits_{i = 1}^{n} {\frac{1}{2}y_{i} y_{i + 1} \sin \theta } } \\ {C = \sum\nolimits_{i = 1}^{n} {\sqrt {{y_{i}}^{2} + {y_{i + 1}}^{2} - 2y_{i} y_{i + 1} \cos \theta } } } \\ \end{array} } \right. $$
(12)

Where \( \theta \) is angle between indicatoraxes, \( \theta = 2\pi /n \). Area Arepresents the level of object being evaluated, while C reflects the balanced development of each index.

3.2 Improvement of Radar Chart

A fatal deficiency in traditional radar chart comprehensive evaluation method is that there is a lack of consideration of indicator weight and correlation between them. It is apparent in Eq. (12) that there are two factors affecting the results of radar chart comprehensive evaluation, i.e. indicator values and angles between indicator axes. Therefore, two points of improvement were conducted in this paper from the two aspects mentioned above, which will make radar chart comprehensive evaluation model more suitable for situations where strong mutual coupling exists.

Indicator Axis Value Integrated with Weight.

For conventional radar chart, the evaluation value (or state value) of each indicator is used as indicator axis value immediately without consideration of indicator weight. In this paper, the indicator weight to the power of indicator evaluation value, i.e. \( {x_{i}}^{{w_{i}}} \), is used as the indicator axis value. Thus the area formula in Eq. (12) will be transformed as follows.

$$ A = \sum\nolimits_{i = 1}^{n} {\frac{1}{2}{x_{i}}^{{{{w_{i} }} }} \cdot {x_{i + 1}}^{{{{w_{i + 1}}} }} \sin \theta_{i} } $$
(13)

Formula (13) is actually a combination of linear weighted model and geometric weighted model, which has certain advantages of both.

Determination of \( \theta_{i} \).

The angles between indicator axes in a conventional radar chartare usually the same, i.e. if there are n indicators, the axis angle will be 2π/n. Correlation coefficients \( r_{i} \) between adjacent indicators are utilized as a basis for the determination of corresponding axis angle \( \theta_{i} \). The calculation formula are as follows.

$$ k_{i} = {\mkern 1mu} \left\{ {\begin{array}{*{20}c} {\sqrt {1 - r_{i}^{2} ,} } & {(0 \le r_{i} \le 1)} \\ {\sqrt {1 - r_{i}^{2} ,} } & {( - 1 \le r_{i} < 0)} \\ \end{array} } \right. $$
(14)
$$ \theta_{i} = 2\pi \cdot k_{i} /\sum\nolimits_{j = 1}^{n} {k_{j} } $$
(15)

Thus the angle between indicator axes will no longer be the average, but determined by correlation coefficients between adjacent indicators. Therefore, the greater the correlation between two indicators, the smaller the angle will be. As a result, their contribution to the composite indicator will be smaller, accordingly. Hence, the improved radar chart is no longer a regular polygon, but similar to the shape shown in Fig. 2. The evaluation value of indicator needs to be normalized, and the negative indicator (negatively correlated with composite indicator) should be transformed into a positive indicator (positively correlated with composite indicator) previously before determining their correlation.

Fig. 2.
figure 2

Example form of improved Radar Chart

Full size image

There are two methods recommended for determining the correlation coefficient \( r_{i} \) in this paper.

  1. 1.

    For the case that there exist multiple objects, supposing p objects, to be evaluated, employing the indicator set \( \{ u_{1} ,u_{2} , \ldots ,u_{n} \} \) to evaluate these objects respectively, then a \( p \times n \)-order evaluation matrix \( \varvec{X} \) can be obtained.

$$ \varvec{X} = [\varvec{X}_{{\mathbf{1}}} ,\varvec{X}_{{\mathbf{2}}} , \ldots ,\varvec{X}_{\varvec{n}} ] = \left[ {\begin{array}{*{20}c} {x_{11} } & {x_{12} } & \ldots & {x_{1n} } \\ {x_{21} } & {x_{22} } & \ldots & {x_{2n} } \\ \vdots & \vdots & \ddots & \vdots \\ {x_{p1} } & {x_{p2} } & \ldots & {x_{pn} } \\ \end{array} } \right] $$
(16)

Where \( x_{ij} \) represents the evaluation value of the i-th object, while \( \varvec{X}_{\varvec{j}} \) denotes the evaluation vector of the j-th indicator on the p objects. Thus, the Pearson correlation coefficient can be calculated and used to represent the correlation between indicators.

$$ r_{{\varvec{ij}}} = \frac{{{\text{Cov(}}X_{i} , { }X_{j} )}}{{\sqrt {D(X_{i} )} \sqrt {D(X_{j} )} }} $$
(17)

Where \( {\text{Cov}}\left( {X_{i} ,X_{j} } \right) \) is the covariance between \( \varvec{X}_{\varvec{i}} \) and \( \varvec{X}_{\varvec{j}} \), \( {\text{D}}\left( {X_{i} } \right) \) and \( {\text{D}}\left( {X_{j} } \right) \) are the variance of \( X_{i} \) and \( X_{j} \) respectively.

  1. 2.

    For the case of only one evaluation object, such as the comprehensive evaluation of a specific alternative during the system design process, expert judgment is advisable for the acquisition of correlation coefficients. The correlation relationship between evaluation indicators may be divided into the following three conditions:

  2. •

    Positive correlation: indicator B will be improved with the improvement of indicator A;

  3. •

    Irrelevant: the improvement of indicator A has no effect on indicator B;

  4. •

    Negative correlation: the improvement of indicator A will worsen indicator B.

When judging the correlations between indicators, the scale presented in Fig. 3 can be served as a reference, and the judgment results can be recorded in the questionnaire shown in Fig. 4 accordingly. In the subsequent data processing, an appropriate correlation coefficient value can be assigned to each level, and finally the mean of expert judgment results will be employed as the correlation coefficient between indicators. This part of work can be carried out simultaneously with the weight consultation process.

Fig. 3.
figure 3

The qualitative grade division of indicator correlation

Full size image
Fig. 4.
figure 4

A questionnaire example for correlation coefficient survey

Full size image

4 Conclusion

In this paper a new approach for comprehensive evaluation of cockpit ergonomic design was proposed. Though there are plenty of researches and applications about comprehensive evaluation/multiple attribute decision making, most of them have a lack of sufficient consideration of the subjectivity of expert judgment and the interdependence between evaluation indicators. The effort of this paper is focused on these two problems.

To mitigate the limitation of individual expert’s knowledge structure and experience, and handle the uncertainty and randomness of subjective judgment, a feedback mechanism was introduced into ORA method for acquiring more reasonable weight coefficients. The new proposed method, named Delphi-ORA method, can promote expert opinions to a consensus through a multi-round consultation. In the multi-round consultation, the average sorting of expert group was utilized as information feedback variable, while the average ordering deviation index based on number of reverse order was defined as loop control variable. In terms of handling the interdependence between indicators, the common method is to ensure the independence between indicators during the development stage of indicator system, or to adjust indicator weights according to their correlations. From the perspective of comprehensive evaluation model, this paper proposed the improved radar chart to incorporate the consideration of interdependence between indicators. In the improved radar chart, angles between indicator axes are determined based on the correlation coefficients of adjacent indicators, and the indicator weight to the power of indicator evaluation value, i.e. \( {x_{i}}^{{w_{i}}} \), is used as the indicator axis value. Thus the improved radar chart comprehensive evaluation model is virtually a conjunction of linear and non-linear comprehensive evaluation model, which has incorporated the correlation between indicators. Therefore, the improved radar chart is very suitable for the comprehensive evaluation of complex and strongly coupled systems such as flight deck.

Therefore, it is easy to find out the weak aspect of system design, and then corresponding improvement can be taken. Although methods in this paper were proposed for the comprehensive evaluation of cockpit ergonomic design, they are also applicable to other areas.