Particle Swarm Optimisation-Support Vector Machine Optimised by Association Rules for Detecting Factors Inducing Heart Diseases

3.1 Feature Selection Based on AR

AR was firstly used to select 14 features from the Cleveland database. During the selection process, only the major itemset for each kind of disease was searched and all the terms in these major itemsets played an important role in the subsequent classification. Therefore, this research just used these items to classify all patients. The major itemsets for various diseases were obtained by using the following algorithms:

• Atherosclerosis

• 1-3-4-7-9-14

• Coronary heart disease

• 2-4-5-7-8-9-11-14

• Rheumatic heart disease

• 1-2-3-6-7-9-11-12-13

• Congenital heart disease

• 2-4-6-8-10-11-13-14

• Myocarditis

• 1-2-4-5-8-9-11-12-13

• Angina

• 1-3-5-7-8-10-11-12

• Arrhythmia

• 3-4-6-7-8-10-12-13

As mentioned above, atherosclerosis can be determined by using features 1, 3, 4, 7, 9, and 14; similarly, other diseases can also be determined based on the above list. Here, experiments were performed using three ARM algorithms, namely the Apriori, Predictive Apriori, and Tertius algorithms.

3.2 SVMs

An SVM [11] is a kind of supervised learning algorithm based on the statistical learning theory. It mainly aims to determine a hyperplane that can optimally segment two classes in the training set. It can realise an improvement of the generalisation ability of learning machines by structure risk minimisation, and the minimisation of empirical risks and confidence intervals. On this basis, it is able to exhibit preferable classification performance in the case of small sample sizes in statistical terms. In addition, SVM is capable of dealing with problems regarding the classification of high-dimensional data. Suppose {xi, yi}i=1N is a training set where x represents the input sampling value and y∈{+1, –1} shows the class label: the hyperplane is defined as w·x+b=0. Here, x denotes the points located on the hyperplane, w determines the direction of the hyperplane, and b represents the deviation of the distance from source to the hyperplane. To find the optimal hyperplane for separating classes, ‖w‖² is required to be minimal under the constraint condition y_i(w·x_i+b)≥1, where i=1, 2, …, n.

Therefore, the following optimisation problem must be solved:

min12∥w∥2

By introducing the positively slack variable ξ_i into the above optimisation problem and permitting the variable to be extended using a non-linear decision surface, a new optimisation problem is presented as follows:

minw,ξ12∥w2∥ + c∑i=1Nξi

Formula (2) is a two programming problem. We used the Lagrange multiplier method to get the Lagrange function

We used the L(w, b, α, ξ, μ) derivative to w, b, ξ_i, and let the derivatives equal to 0

Further,

We used the sequential minimal optimization (SMO) algorithm to obtain the result. In SVM, the classified decision function is converted to

where α_i is the Lagrange multiplier; K(x_i, y_j)=φ(x_i)·φ(x_j) is an asymmetric positive kernel function, which maps data to high-dimensional spaces through use of a non-linear mapping function, φ(x). In this study, the radial basis function (RBF) is defined as K(x_i, x_j)=exp(–‖x_i–x_j‖²/2σ²), where σ is a positive real number and is used in the kernel function.

As presented above, SVMs are essentially binary classifiers. However, identification of many diseases is generally involved in the classification of heart diseases. To solve this problem, we can use multi-class classification strategies. In several recent studies, one-against-all is one of the most popular multi-class classification strategies. Let Ω={w₁, w₂, …, w₆} be the set of heart diseases expected to be classified. Firstly, a binary classification problem that is defined by one disease w_i(i=1, 2, …, 6) that is different from other diseases, for instance, Ω–{w₁}, is solved using a classifier. Then, during the classification process, patients are diagnosed based on the winner-takes-all principle. The detected disease corresponds to the SVM classifier with the highest output.

3.3 Particle Swarm Optimisation

PSO [24] is a stochastic, population-based optimisation technique introduced by Kennedy and Eberhart in 1995. It belongs to the family of swarm intelligence computational techniques and is inspired by social interactions in human beings and animals. Compared with other heuristics, it has fewer parameters to be tuned by the user. Its underlying concepts are very simple. Besides, its coding is very easy and provides fast convergence, and requires less computational burden in comparison with most other heuristics. Also, the behaviour of PSO is not highly affected by increases in dimensionality, and is efficient in tackling multi-objectives, multi-modalities, constraints, and discrete/integer variables.

In PSO, particles move towards the personally, and globally, optimal positions by updating on the following equations:

where t represents the tth iteration; C₁ and C₂ are learning factors that are arbitrary positive numbers between 0 and 1, and that are normally distributed. In the literature, setting C₁=C₂=2 has been proposed as a generally acceptable setting for most problems (Ozcan and Mohan, 1999) [19] and is widely used in practical applications of PSO. w denotes the inertia weight coefficient. A comparative study has been implemented by Han et al. (2010) [8] on different types of inertia weight, and those linearly decreasing along with simulated annealing type are introduced as the best ones. Taking into account both simplicity and efficiency, linearly decreasing inertia weight is the most appropriate method for setting inertia weight and is normally used in PSO applications. The values v_max=0.9 and v_min=0.4 are widely accepted in the literature. x_id and v_id show the position and speed of particle i, respectively; p_id represents the personally optimal position of particle i; while p_gbest is one of the positions of the optimal particle among all particles.

We will introduce some existing research works about PSO, and point out that PSO parameters significantly affect its computational behaviour. Uy et al. (2007) [29] investigated the efficiency of some well-known randomised low-discrepancy sequences (Halton, Sobol, and Faure) for initialising PSO and came to a decision that initialisation with Sobol sequences perform best in PSO [24]. Han et al. (2010) [8] conducted a study about PSO performances with different types of inertia weight, and the study indicated that linearly decreasing inertia weights along with simulated annealing type are the best ones [24]. In the study by Ozcan and Mohan (1999) [19], C₁=C₂=2 has been proved as a generally acceptable setting for most problems, and Guo and Chen (2009) [6] proposed an adaptive variant that assigns specific inertia weight and social acceleration coefficient to each particle while the cognitive acceleration coefficient is kept constant [24].

To deal with discrete problems using PSO, Zhang and Xie used the rounding off method; that is, discrete/integer variables are rounded off to the nearest discrete/integer value during the course of optimisation, and this method does not affect the accuracy obviously [25]. Kennedy and Eberhart transformed the optimisation problem into a binary problem. They use the sigmoid function

s(Vid) = 11 + exp(−Vid),

then generated a random number r in [0,1] and updated the position of particle

Xid(t + 1) = {1(r < s(Vid))0(r ≥ s(Vid)).

Nema et al. (2008) [17] proposed a PSO variant that is hybridised with the branch and bound (BB) algorithm [25]. By this hybridisation, the variant has the advantages of PSO’s global search capability and that of BB’s rapid convergence capability. In the paper of Liu et al. (2009) [14], PSO’s conventional flight equations are modified

Xi1(t) = Xi(t) + αVi(t)Xi2(t) = βPMX[Xi1(t), Pi(t)]Xi1(t + 1) = γPMX[Xi2(t), Pli(t)],

where PMX and P_li represent partially matched crossover and particle i’s neighbourhood best. In the paper of Rezaee Jordehi [23], a novel adaptive PSO method is proposed by adding an adaptive mutation mechanism and a dynamic inertia weight into the conventional PSO method.

3.4 The Training Process

The training process of the proposed algorithm is as follows:

1. Three AR algorithms – Apriori, Predictive Apriori, and Tertius, are used to respectively screen the features relevant to heart diseases from Cleveland data sets. The data sets are then extracted based on the screened features.

2. The parameters for PSO are proposed for PSO-SVMs. Based on the parameters C and ξ_i of SVM, two-dimensional particle swarm is established and each particle in the swarm is then initialised.

3. The parameters C and ξ_i are used to calculate the w vector under the constraint of Formula (2); the solution involving the w vector is then substituted into Formula (3).

4. By integrating the real results of the training samples, classified/total (function M) is used as the fitness function in the PSO algorithm, where total and classified represent the numbers of training samples and accurately classified samples, respectively. If M(Xi)>M(Pi), the optimal position of each particle is updated as Xi; while if M(Pi)>M(Pg), the optimal global position is set as Pg=Pi. Formulae (4) and (5) are used to update the speed and position of each particle.

5. Steps 3 and 4 are repeated until the maximum number of iterations is reached, or the termination conditions are satisfied, to solve the optimised parameters C and ξ_i, as well as for the w vector.

For PSO, too few particles prompt the algorithm to get trapped in local optima, while too many particles slow down the algorithm. We will adjust the training data size based on the AR-selected properties.

4.1 AR Mining Based on Health and Illness Rules

In the first group of experiments, the data are classified into two classes, namely, those of patients and those of healthy people. On this basis, three prevalent ARM algorithms, namely, the Apriori, Predictive Apriori, and Tertius algorithms, are used to select rules with confidence of >90%, confirmation degree of >99%, and precision of confirmation degree of >79%. If there are many such rules, this research merely considers those rules containing “illness” or “health” on their right-hand side (RHS); nevertheless, if no such rule is found, only the rules including “illness” or “health” on their left-hand side (LHS) are considered. The experimental results are shown in Tables 1–3.

As shown in Table 1, four-fifths of the “health” rules are obtained from females, indicating that females are less likely to suffer from coronary heart disease according to this database. In addition, angina (one type of chest pain) induced by exercise being false as an index indicating good health, has nothing to do with gender as these rules have appeared in all LHS rules with high confidence. Moreover, the number of colour vessels equalling zero and thal (a kind of heart state) being normal also represent good health. On the other hand, the mining of “illness” rules suggests that asymptomatic chest pain and thal, being reversible defects, show that a person may be ill; two rules with high confidence including these two factors are found in LHS.

As can be seen in Table 2, differing from the Apriori algorithm, which is based on confidence, the Predictive Apriori algorithm selects rules according to accuracy. Similar to the results obtained with the Apriori algorithm, most “health” rules are extracted from females. However, the factors in the LHS are quite different: angina induced by exercise being false and thal being normal are once again confirmed as indices of good health. Likewise, the range of the maximum heart rate being (149.6, 175.8), the number of colour vessels being zero, and chest pain not related to angina (notang) are proven to be factors indicating good health. Moreover, the “illness” rules, two of which are acquired from males, are obviously different: the slope of the ST segment being flat, age being >48 years, and the existence of colour vessels are shown to be risk indices for heart disease and these factors exist in at least two rules.

The data in Table 3 show that thal being normal is an indicator of good health, and other attributes including the maximum heart rate and cholesterol level are also factors influencing health. The results also indicate that old peak, which represents the ST segment depression caused by exercise, compared with relaxation, being >2.48 can make people susceptible to disease. Old peak appears in the RHS of all “illness” rules. Meanwhile, chest pain and angina induced by exercise are also factors inducing disease.

On the whole, the rules generated by these three algorithms all show that females are less susceptible to heart disease. This discovery is further studied in the next section. Among these three algorithms, the Apriori algorithm operates the fastest, and the rules generated by this algorithm form a pattern. Therefore, it is also used to explore ARs in the second group of experiments.

4.2 Comparison and Analysis

This section compares the classification performance of the proposed algorithm with those of other advanced algorithms, including the algorithm combining an artificial neutral network and fuzzy logic (ANN-FL) [15], that integrating ARs and a neural network (AR-NN) [5], and a feature selection-based SVM (FS-SVM) [27]. In addition, this work classifies seven diseases, namely, atherosclerosis, coronary heart disease, rheumatic heart disease, congenital heart disease, myocarditis, angina, and arrhythmia. The performance of the algorithms is determined by the calculated values of specificity, sensitivity, and overall classification accuracy. Thereinto, the specificity refers to the ratio of the number of true negative decisions to the number of actual negative cases; sensitivity means the ratio of the number of true positive decisions to that of the actual positive cases, while the overall classification accuracy is the ratio of the number of accurate decisions to the total number of cases. The results obtained by the proposed AR-PSO-SVM algorithm are acquired by averaging the statistical results of the three ARM algorithms. When the positive predictions of the algorithm are consistent with those of the doctors, they are proven to be correct positive decisions; when they are inconsistent with those of the doctors, they are shown to be true negative decisions. The statistical parameters for each classifier are summarised in Table 4.

As presented in Table 4, these four algorithms show similar sensitivity and specificity for certain diseases. For instance, their specificities to coronary heart disease, their sensitivities and specificities to angina, and their specificities to atherosclerosis, are all 100%. However, on the whole, the sensitivity, specificity, and overall classification accuracy of the proposed algorithm are not lower than those of the other three algorithms. For example, in terms of rheumatic heart disease, the sensitivity and specificity of the proposed algorithm are 2.32% and 8.71%, higher than those of the FS-SVM algorithm, respectively.

By comparing the overall classification accuracies of these algorithms, it can be seen that the accuracy of the proposed AR-PSO-SVM is 1.67% higher than that of ANN-FL, 4.35% higher than that of AR-NN, and 2.94% higher than that of FS-SVM. In addition, in the comparison of the total time cost for classification, the proposed algorithm costs 36 s less than ANN-FL, 61 s shorter than AR-NN, and 39 s less than FS-SVM. When promising to improve sensitivity, specificity, and overall classification accuracy, the proposed algorithm can still significantly reduce the total time cost for classification; thereby, it is deemed to be both reliable and effective.