Parallel Multi-Class Contour Preserving Classification

1 Introduction

Feed-forward neural network (FFNN) [8, 11, 12] is a wildly used artificial neural network that maps a set of input instances onto a set of outputs. It functions by placing hyper-planes at the decision boundary between consecutive classes of data to partition the data belonging to different classes. It is known that the instances located at the decision boundary between consecutive classes of data generally have more impact on the placement of the hyper-planes than the instances located far away from the decision boundary. However, occasionally, the placement of the hyper-planes may not preserve the concave surface (curves inward) and convex surface (bulges outward) of the data correctly when the space between consecutive classes of data is large. This can adversely affect the levels of classification accuracy.

Consider a problem in Figure 1A, a two-dimensional three-class synthetic data set comprising three classes of instances that are separable by a two-neuron non-linear classifier is presented. When a two-layer FFNN having two hidden neurons is applied, it possibly learns to classify this problem as shown in Figure 1B. Lines 1 and 2 are the classifying hyper-plane that represents the two hidden neurons. The shape or distribution model of the problem is not correctly preserved. When a two-layer FFNN having four hidden neurons is applied, a possible solution may be as shown in Figure 1B. Lines 3, 4, 5, and 6 are the classifying hyper-plane that represents the four hidden neurons. However, applying a typical variation of back-propagation learning algorithm to learn those instances will normally converge to the solution similar to what shown in Figure 1C. The classifier may classify the data inaccurately, especially when the instances are located at the decision boundary between consecutive classes of data because it does not recognize the concave surface and convex surface of the data correctly.

Figure 1:

A comparison among possible classifications: (A) A two-dimensional three-class problem. (B) Possible classification with a two-layer perceptron with two hidden neurons. (C) Possible classification with a two-layer perceptron with four hidden neurons. (D) Typical result of four-hidden-neuron network trained by a variation of back-propagation algorithm [5].

Serial multi-class contour preserving classification [5] is a technique that can improve the representation of the contour of the multi-class data, a set of data having more than two classes, to improve the levels of classification accuracy for FFNN. The algorithm generates two types of multi-class outpost vector (MCOV): fundamental multi-class outpost vector (FMCOV) and additional multi-class outpost vector (AMCOV), from the instances at halfway between consecutive classes of data. Both MCOVs are inserted into the data set to narrow the space between consecutive classes of data to improve the representation of the contour of the data and assist the FFNN to classify the data more accurately. The technique has been significantly improved in [7] to reduce the number of FMCOVs and AMCOVs by maintaining only the FMCOVs and AMCOVs located at the decision boundary between consecutive classes of data based on the Fuangkhon boundary [6].

This article presents an augmentation of the serial multi-class contour preserving classification [5] that overcomes its time deficiency by utilizing thread-level parallelism to support parallel computing on multi-processor or multi-core system. The parallel multi-class contour preserving classification distributes the data set and the processing of the FMCOV and AMCOV generators to be operated on available threads to increase the CPU utilization and the speedup factors of the FMCOV and AMCOV generators. The technique has been carefully designed to avoid data dependency issue. The experiments were conducted on both synthetic and real-world data sets. The experimental results confirm that the speedup factor of the parallel multi-class contour preserving classification over the serial multi-class contour preserving classification is proportional to the number of available processors or processor cores. In other words, the speedup factor increases as the number of available processors or processor cores increases.

The article is organized as follows. Section 2 describes research works related to the serial multi-class contour preserving classification. Section 3 briefly introduces the serial multi-class contour preserving classification. Section 4 presents the implementation of the parallel multi-class contour preserving classification. Section 5 shows the runtime complexity of the parallel multi-class contour preserving classification. Section 6 presents the experimental results. Section 7 presents the conclusion.

2 Related Works

Tanprasert et al. [14] proposed a contour preserving classification technique, called outpost vector model, to preserve the shape or distribution model of two-class data to improve the robustness and weight fault tolerance of a neural network applied with a linearly separable problem. The technique augments a set of original instances with a set of outpost vectors that are generated from the original instances and are located at the boundary between both classes of data. As a result, the learning process is indirectly biased toward distributing classification workload around the set of hidden neurons, thereby forcing the network to perform nonlinear classification. Outpost vectors have some similarities to support vectors [3]. Nevertheless, they are synthesized rather than being selected from a set of original instances. The technique was found in [4] to improve the level of classification accuracy of two-class data. Mongkonsirivatana [10] proposed an improved version of the contour preserving classification technique [14], based on boundary detection, that can reduce the number of instances used to generate outpost vectors. However, determination of the appropriate value of the key parameter is not defined. As a result, the outpost vectors generated from the selected instances only correspond to the boundary of the shape or distribution model of those selected instances. Tanprasert and Kripruksawan [13] proposed an alternative to preserve the contour of the old data for a multi-layer perceptron network, called decayed prior sampling (DPS), that uses subtractive clustering [2]. The technique was designed for adaptive learning. However, the bounded range defined by the average distance of each cluster center will preserve the contour of the old data in a cycle shape only. In addition, the output value of synthesized vectors is defined by the output value of the cluster center, rather than being defined by the classification knowledge of the supervised neural network from the previous training. Kaitikunkajorn and Tanprasert [9] proposed an improved synthesis process of DPS algorithm, called new decayed prior sampling (NDPS), that solves drawbacks in [13] by calculating the bounded range of each feature in the cluster from the average error of each feature in the cluster, instead of the average distance of each cluster center, to maintain the contour of old data. In addition, the technique feeds the synthesized vector into the neural network to get the output value that will be used as the output value of the synthesized vector instead of using the output value of the cluster center. Tanprasert et al. [15] proposed an improved version of DPS algorithm for multi-layer perceptron network, called modified decayed prior sampling (MDPS). The algorithm allows the existing knowledge to age out in slow rate (adaptive) as a neural network is gradually retrained with consecutive sets of new instances, resembling the change of application locality under a consistent environment. It utilizes the outpost vector model [14] rather than subtractive clustering [13] to clarify the boundary between two classes of instances to assist the neural network to classify the data more accurately. The experimental results confirm that MDPS yields higher levels of generalization than NDPS [9]. However, the technique only maintains the contour of new data due to the fact that it generates outpost vectors from new instances only. As a result, the contour of old data is not maintained accurately. It is beneficial to include outpost vectors generated from old data in the final data set to maintain the contour of old data.

3 Serial Multi-class Contour Preserving Classification

The serial multi-class contour preserving classification [5] is a technique that can improve the representation of the contour of the multi-class data, a set of data having more than two classes, to improve the levels of classification accuracy for FFNN. It generates two types of MCOV, FMCOV and AMCOV, from all original instances at the decision boundary between two consecutive classes of data to improve the representation of the contour of the data and to assist the FFNN to classify a linearly separable problem “nonlinearly”. It is an augmented version of the original contour preserving classification [14] that supports multi-class data. The following are the characteristics of FMCOV and AMCOV.

FMCOV is a synthesized vector that is used to declare the decision boundary of the territory of an instance of one class, let us presume an instance i of class A (denoted by A_i ) against an instance of any other class and an instance j of class X (denoted by X_j ), which has smallest Euclidean distance to A_i . It is placed at the boundary of A_i ’s territory in the direction of X_j . X_j is designated as a paired vector of A_i (denoted by ϕ(i)) and mathematically described in Eq. (1).

where i and j are instances and A and X are sets of instances.

A space between the boundary of the center of an instance and the center of its paired vector (denoted by κ) can be inserted to define the space between an FMCOV and an AMCOV of different class. It is intended to provide a small clearance for placing the hyper-planes. It is mathematically described in Eq. (2).

The location of dimension s of an FMCOV of instance i (denoted by o(i)_s ) is mathematically described in Eq. (3).

where i is an instance, i_s is the value of the dimension s of an instance i, ϕ(i)_s is the value of the dimension s of a paired vector of an instance i, and d is the number of dimensions of an instance.

An FMCOV of instance i (denoted by o(i)) is mathematically described in Eq. (4). It is placed at the boundary of A_i ’s territory in the direction of X_j .

where i is an instance and d is the number of dimensions of an instance.

AMCOV is a synthesized vector that is used to declare the decision boundary of a paired vector of an instance, let us presume a paired vector of instance i of class A (ϕ(i)), against that instance i of class A (A_i ). It is placed at the boundary of ϕ(i)’s territory, called counter-boundary, in the direction of A_i . The radius of each dimension of a paired vector of instance i (denoted by r̅(i)_s ) used to generate an AMCOV is mathematically computed by Eq. (5).

where i is an instance and d is the number of dimensions of an instance.

Hence, an AMCOV of instance i (denoted by o′(i)) is mathematically described in Eq. (6). It is placed at the boundary of ϕ(i)’s territory in the direction of A_i . A space (κ) in Eq. (2) is also used to leave some space between an FMCOV and the AMCOV being generated. Many paired vectors may be generated at the same point in the problem space. To reduce the number of duplicated AMCOVs, an AMCOV of an instance will be generated only when a paired vector of a paired vector of that instance is different from that instance.

where i is an instance, i_s is the value of the dimension s of i, ϕ(i)_s is the value of the dimension s of a paired vector of an instance i, r̅(i)_s is the value of the dimension s of the radius of the territory of an instance i, ϕ(ϕ(i)) is a paired vector of a paired vector of an instance i, and d is the number of dimensions of an instance.

The concept of three-class outpost vector is illustrated in Figure 2. There are three classes of data designated as class A, B, and C. To find the territory of each instance, each instance is modeled to span its territory as a circle (sphere in case of three-dimensional space or hyper-sphere in case of more-dimensional space) until the territories collide with another. The territory of instance k of class A, denoted by A_k , is found by locating the instance in any other class that is nearest to A_k . In this case, B^*(A_k ) of class B is nearest to A_k and referred to as A_k ’s pair. Then, the territory of A_k is declared as halfway between A_k and B^*(A_k ). Consequently, the radius of A_k ’s territory is set at half of the distance between A_k and B^*(A_k ). This is to guarantee that if B^*(A_k ) sets its territory using the same radius, then the distance from the hyper-plane to either A_k or B^*(A_k ) will be at maximum. A_k then places its FMCOV against B^*(A_k ) at the decision boundary of A_k ’s territory. The territories of B^*(A_k ) of class B, C^*(B^*(A_k )) of class C, A_j of class A, and B^*(A_j ) of class B are also found by the same method done with A_k of class A. After that, AMCOVs will then be generated from all instances as well. The AMCOV of B^*(A_k ) of class B against A_k of class A is placed at the decision boundary of B^*(A_k )’s territory in the direction of A_k . The AMCOV of C^*(B^*(A_k )) of class C against B^*(A_k ) of class B is placed at the decision boundary of C^*(B^*(A_k ))’s territory in the direction of B^*(A_k ). The AMCOV of A_j of class A against C^*(B^*(A_k )) of class C is placed at the decision boundary of A_j ’s territory in the direction of C^*(B^*(A_k )). The AMCOV of B^*(A_j ) of class B against A_j of class A is placed at the decision boundary of B^*(A_j )’s territory in the direction of A_j .

Figure 2:

FMCOVs, AMCOVs, and Instance’s Territory in a Two-Dimensional Three-Class Problem.

In Figure 2, it is necessary to clarify the differences between the boundary of an instance and the counter-boundary of an instance.

• The boundary of an instance is a circle (sphere in case of three-dimensional space or hyper-sphere in case of more-dimensional space) that identifies the outermost territory of that instance for placing its FMCOV and AMCOV.

  • An FMCOV will be synthesized on this boundary between A_i and ϕ(i) on the territory of A_i .

  • An AMCOV will be synthesized on this boundary between A_i and ϕ(i) on the territory of ϕ(i).

• The counter-boundary of a paired vector identifies the outermost territory of the paired vector against the outermost territory of an instance in the opposite direction. It is not used to place any MCOV but to visualize the boundary of a paired vector against the boundary of an instance in the opposite direction.

Figure 3 illustrates how FMCOVs and AMCOVs are integrated to produce the final class boundary of the data set. Figure 3A illustrates a non-overlapping four-class synthetic data set comprising four sine waves representing four classes of data, designated as red, green, blue, and magenta points in a two-dimensional Euclidean space and determined by two coordinates, x and y. Figure 3B illustrates the synthesized FMCOVs in solid color at the new class boundary between consecutive classes of data. It is noticeable that the FMCOVs shift the original class boundary of one class toward the opposite class. They reduce the space between consecutive classes of data to assist the FFNN to place the hyper-plane in such a way that the contour of the data better preserved. However, some areas on the new class boundary may have very low population of FMCOVs. This area still allows the FFNN to place the hyper-plane that might not preserve the contour of the data accurately, leading to a higher misclassification rate. This is why it is time AMCOVs to come into play. Figure 3C illustrates the synthesized AMCOVs in solid color at the new class boundary between consecutive classes of data. AMCOVs also shift the original class boundary of one class toward the opposite class. They also reduce the space between consecutive classes of data. AMCOVs occasionally fill the space where the population of FMCOVs is low, and vice versa. As a result, the integration of FMCOVs and AMCOVs helps strengthen the new class boundary to ensure the maximum assistance toward the hyper-plane placement process.

Figure 3:

A comparison among various final training sets: (A) An original data set. (B) An original data set with synthesized FMCOVs. (C) An original data set with synthesized AMCOVs. (D) An original data set with synthesized FMCOVs and AMCOVs [6].

4 Parallel Multi-Class Contour Preserving Classification

This section presents the parallel multi-class contour preserving classification, which is an improved version of the serial multi-class contour preserving classification [5], that utilizes thread-level parallelism to offload the processing of the FMCOV generation algorithm and the AMCOV generation algorithm from just a single processor to all available processors or processor cores. This will significantly reduce the total execution time especially on a shared-memory multi-core processor computer. The parallel FMCOV generation algorithm (FMCOV generator) is presented in Algorithm 1.

• In Line 1, two global inputs are defined: the original data set (denoted by T) and the number of available threads (denoted by Tr). Tr should be no larger than the number of thread that the computer can execute concurrently; otherwise, some thread will be idle.

• In Line 2, a global output is defined: a set of FMCOVs (denoted by TF).

• In Line 3, this parallel for-loop offloads the processing of the FMCOV generator from a single thread to all available threads as defined by Tr in Line 1.

• In Line 4, the original data set (T) is divided into Tr partial data sets. Each partial data set (T_t ) is exclusively processed by one and only one thread (t). This for-loop generates FMCOVs (denoted by o(i)) from the partial data set (T_t ) of thread (t) and stores them in the thread (t)’s unshared local subset (S_tf ).

• In Line 5, the algorithm identifies a paired instance of i (denoted by ϕ(i)), which is an instance in any class other than the class of i and has a minimum Euclidean distance to i.

• In Line 6, the FMCOV of i (o(i)) is generated at almost halfway between i and ϕ(i) determined by κ at the boundary of i on the territory of i in the direction of ϕ(i).

• In Line 7, the FMCOV (o(i)) is added into thread (t)’s unshared local subset (S_tf ).

• In Line 8, this is the end of for-loop in Line 4. All FMCOVs (o(i)) of a partial data set (T_t ) are added into thread (t)’s unshared local subset (S_tf ).

• In Line 9, all FMCOVs (o(i)) in thread (t)’s unshared local subset (S_tf ) are added into a set of FMCOVs (TF).

• In Line 10, this is the end of parallel for-loop in Line 3. The FMCOVs in the unshared local subset (S_tf ) from all threads are added into the set of FMCOVs (TF).

The parallel AMCOV generation algorithm (AMCOV generator) is presented in Algorithm 2.

• In Line 1, two global inputs are defined: the original data set (denoted by T) and the number of available threads (denoted by Tr). Tr should be no larger than the number of thread that the computer can execute concurrently; otherwise, some thread will be idle.

• In Line 2, a global output is defined: a set of AMCOVs (denoted by TA).

• In Line 3, this parallel for-loop offloads the processing of the AMCOV generator from a single thread to all available threads as defined by Tr in Line 1.

• In Line 4, the original data set (T) is divided into Tr partial data sets. Each partial data set (T_t ) is exclusively processed by one and only one thread (t). This for-loop generates AMCOVs (denoted by o′(i)) from the partial data set (T_t ) of thread (t) and stores them in the thread (t)’s unshared local subset (S_ta ).

• In Line 5, the algorithm verifies that a paired instance of a paired instance of i (denoted by ϕ(ϕ(i))) is not the same as i before generating AMCOV of i (o′(i)).

• In Line 6, the AMCOV of i (o′(i)) is generated at almost halfway between i and ϕ(i) determined by κ at the boundary of ϕ(i) on the territory of ϕ(i) in the direction of i.

• In Line 7, the AMCOV (o′(i)) is added into thread (t)’s unshared local subset (S_ta ).

• In Line 8, this is the end of a paired instance verification of the AMCOV generator in Line 5.

• In Line 9, this is the end of for-loop in Line 4. All AMCOVs (o′(i)) of a partial data set (T_t ) are added into thread (t)’s unshared local subset (S_ta ).

• In Line 10, all AMCOVs (o′(i)) in thread (t)’s unshared local subset (S_ta ) are added into a set of AMCOVs (TA).

• In Line 11, this is the end of parallel for-loop in Line 3. The AMCOVs in the unshared local subset (S_ta ) from all threads are added into the set of AMCOVs (TA).

After a set of FMCOVs (TF) from Algorithm 1 and a set of AMCOVs (TA) from Algorithm 2 are completely generated, they are combined by UNION to form a final training set. In practice, a set of FMCOVs (TF) must be generated prior to a set of AMCOVs (TA). The MCOV generation algorithm is divided into two algorithms, FMCOV generation algorithm and AMCOV generation algorithm (as shown in Algorithms 1 and 2), for ease of understanding.

5 Runtime Complexity

The runtime complexity of parallel MCOV generation algorithm (T_pmcov(n)) comprises the runtime complexity of parallel FMCOV generation algorithm (T_pfmcov(n)) in Algorithm 1 and the runtime complexity of parallel AMCOV generation algorithm (T_pamcov(n)) in Algorithm 2 as described in Eq. (7).

where n is the number of instances in all classes.

The FMCOVs and AMCOVs are generated from the algorithms presented in Algorithms 1 and 2 using the Euclidean distance function as described in Eq. (8) on a set of instances. The runtime complexity of the Euclidean distance is described in Eq. (9).

where p and q are two points in Euclidean space, p_i and q_i are the value of the feature i of instances p and q, respectively, and d is the number of features of the point.

where p and q are two points in Euclidean space, p_i and q_i are the value of the feature i of instances p and q, respectively, d is the number of features of the point, and z is a constant.

The runtime complexity of the parallel FMCOV generation algorithm (T_pfmcov(n)) is described in Eq. (10).

where n_z is the number of instances in class z, n is the number of instances in all classes, c is the number of classes of data, d is the number of features of an instance, and e is the number of processors or processor cores.

The runtime complexity of the parallel AMCOV generation algorithm (T_pamcov(n)) is described in Eq. (11).

where n_z is the number of instances in class z, n is the number of instances in all classes, c is the number of classes of data, d is the number of features of an instance, and e is the number of processors or processor cores.

As a result, the runtime complexity of the parallel MCOV generation algorithm (T_pmcov(n)) described in Eq. (7) is formulated as Eq. (12).

where n is the number of instances in all classes.

Although the runtime complexity of the parallel MCOV generation algorithm is a linear equation, the actual amount of execution time spent to generate all MCOVs is inversely proportional to the number of available processors or processor cores as demonstrated in Section 6. In other words, the execution time reduces as the number of available processors or processor cores increases.

6 Experimental Results

The experiments were conducted to present an improvement of the CPU utilization and the speedup factors of the parallel FMCOV generation algorithm over the serial FMCOV generation algorithm and the parallel AMCOV generation algorithm over the serial AMCOV generation algorithm. All algorithms were implemented in Microsoft Visual C# 2013 using the following libraries:

• System.Threading.Tasks was required for “parfor” parallel for-loop.

• System.Diagnostics was required for “stopwatch” execution time measurement.

A 1-thread process was used to evaluate the CPU utilization and the execution time of the original serial MCOV generation algorithm. A 2-thread process, a 4-thread process, an 8-thread process, and a 16-thread process were used to evaluate the CPU utilization and the execution time of the parallel MCOV generation algorithm. The machine used to conduct the experiments was powered by Intel^® Core i7 4770k processor having four processor cores with Intel^® Hyper-Threading technology.

The following groups of data sets were used to conduct the experiments. Their characteristics are presented in Table 1.

• A non-overlapping four-class synthetic data set comprises four sine waves representing four classes of data, designated as red, green, blue, and magenta points in a two-dimensional Euclidean space and determined by two coordinates, x and y. This data set consists of 3200 instances from each class of data, which constitutes a total of 12,800 instances.

• Eight highly overlapping two-class synthetic data sets, selected from the ELENA project [16], have a heavy intersection of the class distributions, a high degree of nonlinearity of the class boundaries, and various dimensions of the vectors. These data sets are Clouds, Gaussian 2D, Gaussian 3D, Gaussian 4D, Gaussian 5D, Gaussian 6D, Gaussian 7D, and Gaussian 8D.

• Six real-world data sets, selected from the UCI machine learning repository [1], have a heavy intersection of the class distributions, a high degree of nonlinearity of the class boundaries, and various dimensions of the vectors. These data sets are Adult Income, Statlog (Landsat Satellite), Statlog (Shuttle Landing Control), Forest Cover Level, Pen-Based Recognition of Handwritten Digits, and Optical Recognition of Handwritten Digits.

In terms of CPU utilization, the CPU utilization was record during the execution of the parallel MCOV generation algorithm. The 1-thread process, the 2-thread process, the 4-thread process, the 8-thread process, and the 16-thread process utilized 13%, 25%, 50%, 99%, and 99% CPU times on all data sets, respectively. As mentioned earlier, the Intel^® Core i7 4770k processor with Hyper-Threading technology has only four processor cores and can execute up to 8 threads simultaneously. When the 8-thread process and the 16-thread process ran, the CPU utilization reached its maximum (99% of the CPU utilized). It can be concluded that the CPU utilization can be increased as the number of threads increases.

In terms of execution time, Tables 2 and 3 present the execution time in milliseconds of the parallel FMCOV generation algorithm and the parallel AMCOV generation algorithm on all data sets, respectively. It is noticeable that the reduction of the execution time of the parallel FMCOV generation algorithm and the parallel AMCOV generation algorithm is proportional to the number of threads in the process. In other words, the execution time decreases as the number of available processors or processor cores increases. However, the execution time is slightly reduced when the number of threads is increased from 8 to 16. As mentioned earlier, the Intel^® Core i7 4770k processor has only four processor cores that can operate 4 threads simultaneously. When there are more than 4 threads, Intel^® Hyper-Threading Technology comes into play. This technology allows each processor core to interleave 2 threads operating on the same processor core. As a result, 8 threads can be operated simultaneously. However, this technology shares the floating-point unit (FPU) between the two threads operating on the same processor core. When an FPU of each processor core is being used by one thread to execute the Euclidean distance function or floating-point instruction, another thread assigned to be executed on this processor core will be stalled waiting for the FPU to become available. Moreover, when the number of threads is more than the maximum number of threads that the CPU can handle simultaneously, the overhead of thread switching can adversely affect the execution time of both the parallel FMCOV generation algorithm and the parallel AMCOV generation algorithm. Tables 4 and 5 present the speedup factors of the parallel FMCOV generation algorithm over the serial FMCOV generation algorithm and the parallel AMCOV generation algorithm over the serial AMCOV generation algorithm, respectively, on all data sets. It is intended for clearer understanding.

From the experimental results, it can be concluded that the parallel FMCOV generation algorithm and the parallel AMCOV generation algorithm can utilize thread-level parallelism to enable the parallel computing on multi-processor or multi-core system effectively. The speedup factors of the parallel FMCOV generation algorithm over the serial FMCOV generation algorithm and the parallel AMCOV generation algorithm over the serial AMCOV generation algorithm are proportional to the number of available processors or processor cores. In other words, the speedup factors and the CPU utilization increase as the number of available processors or processor cores increases. The parallel MCOV generation algorithm clearly outperforms the original serial algorithm in terms of CPU utilization and speedup factor.

7 Conclusion

The serial multi-class contour preserving classification helps improve the representation of the contour of the data to improve the levels of classification accuracy for FFNN. The algorithm generates FMCOV and AMCOV to narrow the space between consecutive classes of data to preserve the contour of the data and assist the FFNN to classify the data more accurately. The algorithm has been proven to be able to increase the levels of classification accuracy. However, the algorithm was designed to support only uniprocessor system. This article presents the parallel implementation of the serial multi-class contour preserving classification that overcomes its time deficiency by utilizing thread-level parallelism to support parallel computing on multi-processor or multi-core system. The technique distributes the data set and the processing of the FMCOV and AMCOV generators to be operated on available threads to increase the CPU utilization and the speedup factor of the FMCOV and AMCOV generators. The technique has been carefully designed to avoid data dependency issue. The experiments were conducted on both synthetic and real-world data sets. The experimental results confirm that the speedup factors of the parallel FMCOV generation algorithm over the serial FMCOV generation algorithm and the parallel AMCOV generation algorithm over the serial AMCOV generation algorithm are proportional to the number of available processors or processor cores. In other words, the speedup factor increases as the number of available processors or processor cores increases.