Quality assessment framework to rank software projects

Abstract

Over the last three decades, many quality models have been proposed for software systems. These models employ software metrics to assess different quality attributes. However, without adequate thresholds, it is very hard to associate plausible interpretations with software quality attributes. Many attempts are reported in the literature to identify meaningful thresholds for software metrics. However, these attempts fail to clearly map the proposed thresholds to the assessment of software quality attributes. This paper aims at bridging this gap and provides a methodology for quality assessment models based on software metric thresholds. By doing so, software products can be easily ranked according to specific quality levels. Our methodology defines software metric thresholds to generate ordinal data. Then, the ordinal data is combined with a weighting scheme based on the Pearson correlation coefficient. The resulting weights are assigned to data categories in each software metric. Thanks to these weights, project quality levels are straightforwardly estimated. To assess the effectiveness of our software metric thresholding framework, we carry out an empirical study. The reported results clearly show that the proposed framework has a significant impact on the assessment and evaluation of the software product quality.

Appendix

1.1 Appendix 1

There are three basic interpretations of probability: classical probability, empirical probability, and subjective probability (Bluman 2009).

In the classical probability formula, the probability of any event E is

$$ \,p\left( E \right) = \frac{{Numner{ }\,of\,outcomes{ }\,in\,E}}{Total\,numner\,of\,outcomes\,in\,the\,sample\,space} $$

(13)

This probability uses the sample space (S). It assumes that all outcomes in (S) are equally likely to occur.

In the empirical probability formula, given frequency distribution, the probability of an event being in a given class is

$$ p\left( E \right) = \frac{{frequency{ }\,of\,the\,class}}{{total\,frequencies\,in\,the{ }\,distribution}} $$

(14)

This probability is based on observation. The difference between classical and empirical probability is that empirical probability relies on actual experience to determine the likelihood of outcomes (Bluman 2009).

In this work, we proposed a generalized version of the empirical probability formula to adapt the weights extracted by black-box models (e.g., AI models). These weights provide abstract semantics of the impact of many factors. Instead of considering many factors while building the probabilistic models, we tried to abstract them inside the weights, and then the proposed formula will be applied to those weights to build a probabilistic model. This idea is useful to abstract the complexity and allows building something above something. This will help model a very complex concept. The proposed formula can be defined as follows: Given a weighted sum distribution, the probability of an event being in a given class is

$$ p\left( E \right) = \frac{{the{ }\,weighted\,sum\,of\,{ }the\,class}}{{the\,total{ }\,weighted\,sum\,in\,the\,distribution{ }}} $$

(15)

This probability is also empirical probability and is based on weights identified by observers. Observers usually are AI models that analyze huge amounts of data to study many relations and abstract the impact of many factors as weights. The following examples will give an interpretation of this formula.

Example 1

Given a software project with three entities E = {e₁, e₂, e₃} described in Fig. 7

Fig. 7

Information of the given software project (Color figure online)

• The metric m₁ measures the three entities as (good, good, bad).

• The metric m₂ measures the three entities as (good, bad, bad).

• Table

  Table 14 Frequency distribution (Color table online)

  14 shows the frequency distribution.

1.1.1 What is the probability that the given software project is in a good quality level?

This simply can be solved by the frequency formula presented in Eq. (14) as follows:

The next examples show cases where the weights are involved, and the proposed formula in Eq. (15) is applied.

Example 2

Given a software project with three entities E = {e₁, e₂, e₃} described in Fig. 

Fig. 8

Information of the given software project

8.

• 1. The metric m₁ measures the three entities as (good, good, bad).

• 2. The metric m₂ measures the three entities as (good, bad, bad).

• 3. Considering that the impact of m₂ to identify the quality of the project is twice m₁ where the impact is measured by a black-box model, which gives m₁ a weight (0.2) and m₂ a weight (0.4). The weighted sum distribution and the frequency distribution are displayed in Table

  Table 15 Weighted and frequency distribution (Color table online)

  15.

1.1.2 What is the probability that the given software project is in a good quality level?

As mentioned above, the impact of metric m₂ to identify the quality of the project is twice m₁. This impact is given by these weights m₁ = 0.2 and m₂ = 0.4. While there is a new factor which is the impact of each software metric on identifying the quality of the software, then the proposed formula presented in Eq. (15) is applied to compute the probability that the given software project is of a good quality level.

The following page provides an interpretation of the proposed formula to calculate the probability.

The above example can also be modeled using multiplication rules, addition rules, and conditional probability. While the impact of metric m₂ to identify the quality of the project is twice m₁, then this simply means ${\varvec{p}}({\varvec{m}}_{1} ) = \frac{1}{3},{\text{ and }}{\varvec{p}}({\varvec{m}}_{2} ) = \frac{2}{3}$.

With the use of a tree diagram, the sample space can be determined, as shown in Fig. 

Fig. 9

Probability tree diagram

9. First, assign probabilities to each branch. Next, using the multiplication rule, multiply the probabilities for each branch. Finally, use the addition rule since a good quality level can be identified by metrics m₁ or m₂.

$$ {\varvec{p}}\left( {{\varvec{good}}} \right) = p\left( {{\varvec{m}}_{1} } \right) \cdot p\left( {{\varvec{good}}|{\varvec{m}}_{1} } \right) + p\left( {{\varvec{m}}_{2} } \right) \cdot p\left( {{\varvec{good}}|{\varvec{m}}_{2} } \right) $$

$$ {\varvec{p}}\left( {{\varvec{good}}} \right) = \frac{2}{9} + { }\frac{2}{9} = \frac{4}{9} $$

This shows the same result with comparing to the proposed formula, which provides interpretation. In the next page, we will show another example that helps understand the idea. We will model our problem as a box-color ball problem.

Example 3

In this example, the problem described in example-2 is modeled here as a box-color ball problem. In Fig. 

Fig. 10

Two boxes: Box m1 contains small size balls, m2 contains big size balls

10, the box m₁ contains small balls with 1 red ball and two green balls. Box m₂ contains big balls with two red balls and one green ball. The preference of selecting a box with a big ball is twice of selecting a box with small balls. Assuming that the preference is measured by a black-box model, which gives m₁ a weight (0.2) and m₂ a weight (0.4). This simply means ${\varvec{p}}({\varvec{m}}_{1} ) = \frac{1}{3},{\text{ and }}{\varvec{p}}({\varvec{m}}_{2} ) = \frac{2}{3}$.

1.1.3 Find the probability of selecting a green ball?

With the use of a tree diagram, the sample space can be determined, as shown in Fig. 

Fig. 11

Probability tree diagram

11. First, assign probabilities to each branch. Next, using the multiplication rule, multiply the probabilities for each branch. Finally, use the addition rule since a green ball can be obtained from the box m₁ or box m₂.

$$ {\varvec{p}}\left( {{\varvec{green}}} \right) = p\left( {{\varvec{m}}_{1} } \right) \cdot p\left( {{\varvec{green}}|{\varvec{m}}_{1} } \right) + p\left( {{\varvec{m}}_{2} } \right) \cdot p\left( {{\varvec{green}}|{\varvec{m}}_{2} } \right) $$

$$ {\varvec{p}}\left( {{\varvec{green}}} \right) = \frac{2}{9} + { }\frac{2}{9} = \frac{4}{9} $$

Example 4

Given a software project with three entities E = {e₁, e₂, e₃} described in Fig. 

Fig. 12

Information of the given software project

12.

• 1. The metric m₁ measures the three entities as (good, good, bad).

• 2. The metric m₂ measures measures the three entities as (good, good, bad).

• 3. The metric m₃ measures the three entities as (good, bad, bad).

• 4. Considering that the impact of m₂ to identify the quality of the project is twice m₁, and m₃ is twice m₂ where the impact is measured by a black-box model, which gives m₁ a weight (1.1), m₂ a weight (2.2) and m₃ a weight (4.4). The weighted sum distribution and the frequency distribution are described in Table

  Table 16 Weighted and frequency distribution (Color table online)

  16.

1.1.4 What is the probability that the given software project is in a good quality level?

We can apply the proposed formula in Eq. (15) as follows:

The following page provides an interpretation of how this probability can be calculated using multiplication rules, addition rules, and conditional probability.

The above example can also be modeled as follows:

While the impact of metric m₂ to identify the quality of the project is twice m₁, and m₃ is twice m₂, then this simply means ${\varvec{p}}({\varvec{m}}_{1} ) = \frac{1}{7}$, ${\varvec{p}}({\varvec{m}}_{2} ) = \frac{2}{7}$, and ${\varvec{p}}({\varvec{m}}_{3} ) = \frac{4}{7}$.

With the use of a tree diagram, the sample space can be determined, as shown in Fig. 

Fig. 13

Probability tree diagram

13. First, assign probabilities to each branch. Next, using the multiplication rule, multiply the probabilities for each branch. Finally, use the addition rule since a good quality level can be identified by metrics m₁, m₂ or m₃.

$$ {\varvec{p}}\left( {{\varvec{good}}} \right) = p\left( {{\varvec{m}}_{1} } \right) \cdot p\left( {{\varvec{good}}|{\varvec{m}}_{1} } \right) + p\left( {{\varvec{m}}_{2} } \right) \cdot p\left( {{\varvec{good}}|{\varvec{m}}_{2} } \right) + p\left( {{\varvec{m}}_{3} } \right) \cdot p\left( {{\varvec{good}}|{\varvec{m}}_{3} } \right) $$

$$ {\varvec{p}}\left( {{\varvec{good}}} \right) = \frac{2}{21} + { }\frac{4}{21}{ } + { }\frac{4}{21} = \frac{10}{{21}} = \frac{11}{{23.1}} = 0.48 $$

This shows the same result with comparing to the proposed formula, which provides interpretation.

Example 5

Given a software project with three entities E = {e₁, e₂, e₃} described in Fig. 

Fig. 14

Information of the given software project

14

• 1. The metric m₁ measures the three entities as (bad, good, good).

• 2. The metric m₂₁ measures the three entities as (bad, good, good).

• 3. The metric m₃ measures the three entities as (bad, good, good).

• 4. Considering that the impact of each quality level in each metric is different. Assume that this impact is measured by a black-box model, which gives weights described in Table

  Table 17 Weights identified by black-box model

  17. The weighted sum distribution and the frequency distribution are described in Table

  Table 18 Weighted and frequency distribution (Color table online)

  18.

1.1.5 What is the probability that the given software project is in a good quality level?

We can apply the proposed formula in Eq. (15) as follows:

The following page provides an interpretation of how this probability can be calculated by modeling it as a Box-color ball problem.

The above example can also be modeled as a box-color ball problem. In Fig. 

Fig. 15

Two boxes: Box1 contains small size balls, box2 contains big size balls

15, box1 contains small balls with two red balls and four green balls. Box2 contains big balls with one red ball and two green balls. Assuming that the preference of selecting a box with a big/small ball is measured by a black-box model, which gives box1 a weight (2.2) and box2 a weight (2.2). This simply means ${\mathbf{p}}\left( {{\mathbf{box}}1} \right) = \frac{0.6}{{2.8}},{\text{ and }}{\mathbf{p}}\left( {{\mathbf{box}}2} \right) = \frac{2.2}{{2.8}}$.

1.1.6 Find the probability of selecting a green ball?

With the use of a tree diagram, the sample space can be determined, as shown in Fig. 

Fig. 16

Probability tree diagram

16. First, assign probabilities to each branch. Next, using the multiplication rule, multiply the probabilities for each branch. Finally, use the addition rule since a green ball can be obtained from box1 or box2.

$$ {\varvec{p}}\left( {{\varvec{green}}} \right) = p\left( {box1} \right) \cdot p{\text{(green|}}box1) + p\left( {box2} \right) \cdot p{\text{(green|}}box2) $$

$$ {\varvec{p}}\left( {{\varvec{green}}} \right) = \frac{0.6}{{2.8}} \cdot \frac{2}{3} + \frac{2.2}{{2.8}} \cdot \frac{2}{3} = \frac{2}{3} = 0.67 $$

1.2 Appendix 2

When we deal with feature evaluators, we have two categories which are feature weighting techniques and feature selection techniques.

For feature weighting schema, the concept is to give weight to the given features based on two factors which are relevance and/or redundancy. Relevance is about measuring how much the given feature is relevant to the class (target attribute) by examining the correlation between the given feature and the class. Redundancy is about measuring the redundant features to predict the class by examining the correlations among features. A feature is said to be redundant if one or more of the other features are highly correlated with it (Hall 1999).

For feature selection techniques, the concept is about selecting a subset of the given features. The selection can be achieved with the help of a feature weighting schema. For feature selection techniques, there are some techniques that can be used to remove irrelevant features, some techniques to remove redundant features, and some techniques to remove both irrelevant and redundant features. The purpose of feature selection techniques is to improve performance by removing redundant or/and irrelevant features (Hall 1999). In feature selection techniques, a good subset of features is the one that contains features highly correlated with the class and less correlated with each other (Hall 1999).

While feature weighting schema provides techniques to assign a weight to each feature, the feature selection techniques use these weights along with some selection techniques to select the most useful subset of features.

For example, the correlation-based feature selection technique (CFS) that is proposed by Mark Hall (Hall 1999), uses a feature weighting algorithm called RELIEF. CFS is based on the feature subset evaluation function which is given by Eq. (16). CFS is a simple filter algorithm that ranks feature subsets according to a correlation calculated by RELIEF. CFS is, in fact, a Pearson’s correlation coefficient (Hall 1999).

$$ Ms = \frac{{kr_{cf} }}{{\sqrt {k + k\left( {k - 1} \right)r_{ff} } }} $$

(16)

where $Ms$ is the heuristic “merit” of a feature subset s containing $k$ features, $rcf$ is the mean feature-class correlation $\left( {f{ } \in { }S} \right)$, and $rff$ is the average feature-feature intercorrelation. The numerator of Eq. (16) can be thought of as providing an indication of how much a set of features can predict the class; the denominator of how much redundancy there is among the features.

According to the following example that is extracted from Hall’s thesis (Hall 1999), Table

Table 19 Feature correlations calculated from the "Golf" dataset using Relief. This table is extracted from Hall’s thesis (Hall 1999)

19 shows the feature’s weights that are calculated by RELIEF, between the given feature and the other features and between the given feature and the class. Table

Table 20 A forward selection search using the correlations in Table 2 . This table is extracted from Hall’s thesis (Hall 1999)

20 shows how these weights are utilized by CFS to select the best subset of features.

The search starts with an empty set of features. By applying the evaluation function, subsets in bold show a local improvement with respect to the previous best set.

Relief is a feature weight schema that can be utilized for feature selection problems. Relief is based on a statistical method to calculate the weight of the given feature (Kira and Rendell 1992). It is an algorithm that is inspired by instance-based learning. Relief can detect features that are statistically relevant to the target attribute. It considers nominal features (including Boolean) or numerical features (integer or real). The first version of this algorithm is based on a two-class classification problem where the target attribute (class or concept) has only two values (Kira and Rendell 1992). However, the relief is extended by Kononenko (Kononenko 1994) to be applicable for multi-class classification problems. According to the Relief algorithm (Kira and Rendell 1992), the training dataset (S) is separated into two sets $\left\{ {S + ,{ }S - } \right\}$, instances with positive class value and instances with negative class value (assuming the class attribute has two values, positive and negative). Relief picks a sample with $M$ instances triplets of an instance $X$. The instance $X$. is represented by a vector that is composed of $p$ feature values $\left\{ {x1,{ }x2,{ } \ldots ,{ }xp} \right\}$. where the given feature set is $\left\{ {f1,{ }f2,{ } \ldots ,{ }fp} \right\}$. For each instance $X$, the algorithm picks one of the positive instances $\left\{ {S + } \right\}$. that is closest to $X$., and picks one of the negative instances s $\left\{ {S - } \right\}{ }$. that is closest to $X$.. Relief uses the p-dimensional Euclid diance for selecting the closest instances. Then, it determines which one is a near-hit and which one is a near-miss to the instance $X$. Near-hit is the closest instance to $X$. with same target value, and the Ne-miss is the closest instance to $X$ with different class target value. After that, Relief calls a routine to update the feature weight vector $W$ for every sample triplet and determines the average feature weight vector relevance (of all thfeates to the target concept). $W$. is initialized with zeros: $(0,{ }0,{ } \ldots ,{ }$ a updad using the following formula:

$$ For{ }\,i \,=\,1{ }\,to{ }p{ } $$

$W_{i} = W_{i} { } - diff\left( {x_{i} ,\,near\,Hit_{i} } \right)^{2} { }\, + \,diff\left( {x_{i} ,{ }\,near\,Miss_{i} } \right)^{2} { }$where $diff$. is a function that can be described for nominal feature values as follows:

$$ Diff\left( {x,y} \right) = \left\{ {\begin{array}{*{20}l} 0 \hfill & {if\, x\, and \,y \,are \,the \,same} \hfill \\ {} \hfill & \cdot \hfill \\ 1 \hfill & {if\, x \,and\, y \,are \,different} \hfill \\ \end{array} } \right. $$