Zen-ReqOptimizer: a search-based approach for requirements assignment optimization

Abstract

At early phases of a product development lifecycle of large scale Cyber-Physical Systems (CPSs), a large number of requirements need to be assigned to stakeholders from different organizations or departments of the same organization for review, clarification and checking their conformance to standards and regulations. These requirements have various characteristics such as extents of importance to the organization, complexity, and dependencies between each other, thereby requiring different effort (workload) to review and clarify. While working with our industrial partners in the domain of CPSs, we discovered an optimization problem, where an optimal solution is required for assigning requirements to various stakeholders by maximizing their familiarity to assigned requirements, meanwhile balancing the overall workload of each stakeholder. In this direction, we propose a fitness function that takes into account all the above-mentioned factors to guide a search algorithm to find an optimal solution. As a pilot experiment, we first investigated four commonly applied search algorithms (i.e., GA, (1 + 1) EA, AVM, RS) together with the proposed fitness function and results show that (1 + 1) EA performs significantly better than the other algorithms. Since our optimization problem is multi-objective, we further empirically evaluated the performance of the fitness function with six multi-objective search algorithms (CellDE, MOCell, NSGA-II, PAES, SMPSO, SPEA2) together with (1 + 1) EA (the best in the pilot study) and RS (as the baseline) in terms of finding an optimal solution using an real-world case study and 120 artificial problems of varying complexity. Results show that both for the real-world case study and the artificial problems (1 + 1) EA achieved the best performance for each single objective and NSGA-II achieved the best performance for the overall fitness. NSGA-II has the ability to solve a wide range of problems without having their performance degraded significantly and (1 + 1) EA is not fit for problems with less than 250 requirements Therefore we recommend that, if a project manager is interested in a particular objective then (1 + 1) EA should be used; otherwise, NSGA-II should be applied to obtain optimal solutions when putting the overall fitness as the first priority.

Appendix

In this section, we present results of evaluating the multi-objective search algorithms using Hyper Volume (HV)—a commonly used quality indicator with multi-objective search algorithms. The results are for the full-scale empirical study. Recall that each search algorithm was run 100 times for each problem and generated a Pareto front at each run. We calculated the HV value of each Pareto front.

1.1 Real-world Case Study

We obtained 100 HV values for the real world case study for each algorithm. We conducted the Wilcoxon signed-rank test at the significance level of 0.05 for the HV values and results are presented in Table 19. Algorithms with larger values of HVare desirable; if $\hat {A}_{12}$ is greater than 0.5, it means that algorithm A has a higher chance of obtaining a higher HV than B. An $\hat {A}_{12}$value less than 0.5 means the Algorithm A has lesser chance of obtaining a higher value of HV than B. From Table 19, we can conclude that in terms of HV, NSGA-II obtains significant better results than SPEA2, followed by MOCell, PAES and SMPSO. CellDE obtains the worst. These results are consistent with what we observed from Section 4.2.1.

Table 19 Results of the Vargha and Delaney statistics and the Wilcoxon signed-rank test at the significance level of 0.05 (in terms of HV) – real-world case study

1.2 Artificial Problems

For each of the 120 artificial problems, we conducted the one sample Wilcoxon signed-rank test to compare each pair of the algorithms in terms of HV. Table 20 summarizes the results of the Vargha and Delaney statistic test (with or without the Wilcoxon signed-rank test applied). Without the Wilcoxon signed-rank test, A>B means the number of problems (out of 120) that A is better than B for obtaining a better solution; A<B means the number of problems (out of 120) that A is worse than B for obtaining a better solution; and A = B means the number of problems for which there are no differences between A and B.

Table 20 Results of the Vargha and Delaney statistical test (in terms of HV) - 120 artificial problems (without/with the Wilcoxon signed-rank test)

Results show that for HV, NSGA-II performed significantly better than SPEA2 for 115 problems and SPEA2 performed significantly better than PAES for 72 problem. PAES performed significantly better than MOCell for 65 problems. MOCell performed significantly better than CellDE for 118 problems. CellDE performed significantly better than SMPSO for 55 problems and significantly worse than SMPSO for 30 problems. SMPSO and CellDE had no significant difference for 35 problems. Based on the results, we can conclude that in terms of HV, NSGA-II achieves the best performance, followed by SPEA2, PAES and MOCell. CellDE and SMPSO share similar performance, where both performed worse than the other search algorithms. In terms of HV, for all the 120 artificial problems, results are similar to what have been reported in Section 4.2.2.