J³ Model: A novel framework for improved Modified Condition/Decision Coverage analysis

Highlights

• J³ Model is proposed to achieve high MC/DC percentage that shows the improvement of the existing concolic testing technique.

• JPCT is an efficient version of transformation technique as compare to existing technique.

• JCA is much more power full than existing coverage analyser for MC/DC, since it is developed by considering all MC/DC essential requirements.

• Implementation and Experiments are done for forty Java programs, and the results present the improvement in MC/DC.

Abstract

In the real-time systems and safety critical domains, software quality assurance adheres to protocols such as DO-178C standard. Regarding these issues, concolic testing generates test cases that can attain high coverage using an augmented approach based on Modified Condition/Decision Coverage (MC/DC). In this paper, we propose a framework to compute MC/DC percentage for test case generation. To achieve an increase in MC/DC, we transform the input Java program, J, into its transformed version, $\mathrm{J}\prime$, using Java Program Code Transformer (JPCT). Then, we use JCUTE tool to generate test cases. At last, we use Java Coverage Analyzer (JCA) to compute MC/DC percentage. The Java program code transformer adds additional empty nested if-else conditional statements for each decision that causes variation in MC/DC percentage. In later step, these extra conditional statements get stripped-off. This approach resolves some of the bottleneck issues associated with traditional concolic testers. In our experimental study, we have experimented with forty Java programs. We have computed the difference of MC/DC%, for both the scenarios (i.e. with code transformation and without code transformation). Our approach (i.e. with code transformation achieves) 24.09% average increase in MC/DC% over the traditional approach (i.e. without code transformation).

Introduction

Traditional testing techniques, often fail to detect errors or defects in software systems. One of the reasons for failure is that a program may have an enormous number of different execution paths because of the presence of loops and conditional statements. Therefore, it is unlikely for a software tester to create enough test cases manually to detect subtle errors in all the possible execution paths. It is technically a significant challenge to generate a test suite that covers all different paths in an automated manner [22].

To address these issues, CONCOLIC testing has been proposed. CONCOLIC (CONCrete+symbOLIC) [6] testing combines concrete dynamic and static symbolic analysis. This testing technique is used to automatically generate test cases for exploring the possible execution paths of a program [17], [18], [23], [30]. Concolic testing may take a significant amount of time to explore all the execution paths, and this is an issue towards its practical application [8], [11].

Software testing is of two types, one is Black Box testing and another one is White Box testing. Coverage based testing is a white-box testing technique. It includes several testing criteria such as Statement Coverage (SC), Condition Coverage (CC), Branch Coverage (BC), Modified Condition/Decision Coverage (MC/DC), Multiple Condition Coverage (MCC) etc. Though MCC is the strongest criterion but we have chosen MC/DC, because MC/DC generates non-redundant test cases [24]. The process of manually achieving coverage is both very difficult and effort intensive. In accordance with the RTCA/DO-178C standard, coverage-based testing is a measure of acceptability of the requirement-based testing in the context of exercising logical expressions.

Our contribution in this paper is to propose a model to generate test cases that achieve higher MC/DC percentage. In this paper, we propose a framework to compute the MC/DC percentage of Java programs. We have named the proposed framework JPCT JCUTE JCA MODEL (J³ Model). J³ Model consists of mainly three modules: i) Java Program Code Transformer (JPCT), ii) Java Concolic Tester (JCUTE), iii) Java Coverage Analyser (JCA). Out of these three modules, we have proposed and developed two modules – JPCT and JCA, and used an existing open source tool called jCUTE. Our approach integrates these three modules. In our approach, we have developed JPCT to improve the effectiveness and efficiency of the proposed model by generating additional conditional statements. J³ Model overcomes the problems of traditional concolic testing. JCUTE [4] is an open source concolic tester. It takes the non-transformed (original) program and the transformed program as input and generates the test cases as the output. We have developed JCA to compute the MC/DC percentage. This proposed work is the extension of our previous work [12]. In this proposed approach we have improved the technique that is presented in [12], [13], [14]. In the previous work [12], [13], [14], we have used C language to develop the Program Code Transformer (PCT) and Coverage Analyzer (CA) to handle structured programs. PCT was having a number of limitations such as: i) Non-capable of handling larger size programs, ii) The insertion idea of additional if-else statements was based on string matching which poses a constraint on the maximum number of possible combinations of predicates, iii) The Quine McCluskey technique was implemented in C language and, that can only handle up to four variables.

But in our proposed JPCT, we have resolved these aforementioned issues. JPCT is developed in Java language and is capable of handling larger size object-oriented programs. JPCT is developed in a generalized way that can handle all possible combinations for a Sum of Product (SOP) irrespective of number of variables present in the SOP. In JPCT, Quine McCluskey technique is used which is also implemented in Java language. This Java version of Quine McCluskey technique is capable of handling approx. 15 to 18 variables (×4 than older Quine McCluskey technique). In our previous work, we have used a concolic engine called CREST, which supports structured programs (e.g. C programs). Whereas in this proposed work, we have used jCUTE that supports Object-Oriented programs (e.g. Java programs). CREST is non-capable of selecting input values for strings, whereas jCUTE is capable of selecting string values also. In our previous work [12], [13], [14], [15], we have used Coverage Analyzer (CA) which was developed in C language to measure MC/DC%. CA was implemented using two-flag concept. This logic only identifies whether both true_flag and false_flag are TRUE or not. If it is TRUE, then the current condition is included in independently affected conditions set [Please refer to Algorithm for CA in [12]. In this proposed work, we have implemented JCA on the basis of all MC/DC requirements in Java language. JCA is more powerful than the previous version CA to measure MC/DC%. JCA is more scalable than CA, because it can handle large number of conditions in a predicate than CA. Since, CA was developed using C language, so, whenever a C program gets executed to invoke a specific required pre-defined function, we have to include the whole header file, where this function prototype exists. But, JCA is developed using Java language. When we required to invoke a specific pre-defined method, then we can use only the particular pre-defined method of the particular class not the whole package. The advantages of Java language increase the time and memory efficiency.

The rest of the paper is organized as follows: Section 2 discusses some of the basic concepts required to understand the proposed work. Section 3 introduces the proposed J³ Model framework and explains the required steps for implementation. We present the analysis of our experimental results in Section 4. In Section 5, we discuss comparisons of the existing related works with our proposed approach. We report some of the current threats to the validity of our approach in Section 6. Section 7 concludes the paper with some insights into our future work.