1 Introduction

System development more and more rely on models describing the system’s environment, potential interaction as well as its intended behavior. Such models often conceptualize the application domain and are often available in a formal form that can be used for various purposes. Ontologies, which are a formal conceptualization of entities, their interfaces and behaviors, and relationships, describe the knowledge behind such an application domain are more often used for various purposes. For example, in the context of autonomous driving such ontologies have been used for decision making [17], describing traffic situations [4], and navigation [21].

In the context of testing and in particular test suite generation, ontologies provide information about certain entities and their relationships from which we might extract test cases. Let us have a look at simplified text ontology comprising the concepts of Text, Sentence, Delimiter and Word. An instance of a Text is a sequence of sentences, i.e., instances of Sentence followed by a instance of Delimiter. A particular sentence comprises words, which are instances of the concept Word. We depict the relationships between these concepts in Fig. 1. A test case would be a certain instance of Text comprising its parts specified in the ontology. If we want to use such tests for verifying a program like a text processor, we would be interested in obtaining certain combinations of words and delimiters that form a sentence.

One possible way of coming up with such combinations would be the use of combinatorial testing [8] that has been successfully applied for verifying different software applications. Combinatorial testing in its simplest form takes a set of variables representing inputs and parameters together with their domains as input model and generates test cases including certain combinations of variable values. In particular, in combinatorial testing we search for all tests that cover all combinations for any subset of size k of the variables, where k is called the strength of the generated test suite.

Unfortunately, there has been no algorithm reported that takes an ontology as input and which returns a combinatorial test input model (or a combinatorial test suite of strength k) as output. In this paper, we focus on this open challenge and present an algorithm that allows using ontologies as input for the generation of combinatorial test suites. For this purpose, we formalize ontologies and discuss an algorithm that maps those ontologies to combinatorial input models. In addition, we show the underlying functionality of the algorithm using the text ontology as a case study.

Fig. 1.
figure 1

A simplified textual input ontology depicted using UML comprising sentences, which themselves have words and one delimiters.

Full size image

The intention behind this work is to use it for test suite generation of ontologies for autonomous driving, which are currently under development. There the focus is on tests that reveal situations an autonomous vehicle has to deal with under which it might fail. A first ontology comprises more than 200 concepts from which we want to generate test suites used to verify the correct functionality of an autonomous vehicle even in critical situations comprising different combinations of static elements like roads or dynamic elements like other cars of pedestrians.

We structure this paper as follows: First, we discuss related research including mappings of UML diagrams to combinatorial test suites. Afterwards, we formally introduce the concept of ontologies and in order to be self-contained discuss the foundations of combinatorial testing. We further introduce the algorithm for converting ontologies into combinatorial test suites followed by a section where we apply this algorithm on our text ontology. Finally, we conclude the paper.

2 Related Research

Software testing is tedious and expensive yet critical to quality assurance. Designing suitable test cases requires that programmers possess a certain amount of domain knowledge. Therefore, ontology can aid in software testing as they encode domain knowledge in a machine processable format. It provides people and software systems a common shared understanding of knowledge which is easy to organize, maintain and update [13]. Although the study of ontology has become active since the last decade, few works have been done regarding the ontology-based software testing, especially in the field of combinatorial testing (CT). Li et al. [10] developed an ontology to generate user-centric GUI test cases. First, GUI and non-GUI components are captured by reverse engineering techniques. Relations among GUI components are then analyzed to create a GUI ontology and rules for test case generation.

Nguyen et al. [15] proposed an ontology-based test generation framework for multi-agent systems. An ontology that defines content semantics of agent interactions is developed to generate test inputs, guide the input space exploration, and verify messages exchanged between agents. Li and Ma [11] discussed how to generate the parameter setting interface for test atom in spacecraft automatic test based on an ontology knowledge base. The knowledge base represents the information extracted from the test atom and user interface and the rules pre-defined to the mapping relationship between test atom attributes and interface elements.

Satish et al. [18] proposed a rule-based approach to derive CT parameters and values from use case specification (UCS) and UML use case diagrams (UCD). The UCS specifies the requirements in a predefined structure such as the precondition, normal-flow, alternative-flow, and postconditions. The UCD depicts the software under test (SUT) from functionality point of view, and the function is encompassed within a use case in a UCD. The rules are formulated to derive the combinatorial test design model (CTDM) elements from the UCS and UCD parsers. A set combined set of parameters and values are then derived from these rules to provide suggestions to the testers for further elements selection. Before this work, Satish’s team had applied similar strategy to derive CT parameters and values from UML activity diagrams [20] and sequence diagrams [19].

Use case diagrams focus on the system’s functional usage and their interrelationships, dependencies, etc. They are useful for understanding how a feature is used. However, they isolate the details of the system. These details provides information for deriving CT parameters and values. The UML class diagrams, which we use to depict ontology in this paper, directly shows the reflections of the real world as objects (i.e., classes) and their interrelationships. Different use cases identified in use case diagram can turn into classes. In addition, the ontology depicted by UML class diagrams can be represented in hierarchical style which is perfect for generating the target CT model. Therefore, using ontology depicted by class diagrams is more straight forward to structure the entire system in a CT-oriented way.

Moser et al. [12] argued that measurable benefits such as the feasibility of the ontology-based test case generation approach and the cost-benefit potential of ontology-based test case generation approaches (e.g., cost-benefit for a constant number of parameters and for expanding the number of parameters) are essential to decide whether an ontology-based approach and the change costs is acceptable or not. Studies in [1, 14, 22, 23, 25] designed ontologies that focus on the software testing process and the corresponding activities and artifacts. The aim is to manage software testing knowledge so that different knowledge items in software testing are collected, shared, reused and organized.

3 Basic Definitions

In this section, we briefly outline the basic foundations of ontologies used for providing input models, and combinatorial testing used for generating test cases from the given model. In contrast, to ordinary combinatorial input models, the ontology-based models provide more details about relationships between different conceptual entities.

3.1 Ontology

Feilmayr and Wöß [3] stated that “an ontology is a formal, explicit specification of a shared conceptualization that is characterized by high semantic expressiveness required for increased complexity ”. From this informal definition we are able to deduce that ontologies need to describe concepts in a formal way as well as knowledge about these concepts including their relationships. In the following, we introduce a very much simplified definition of ontologies based on concepts restricting relations to composition and inheritance. We use the former relations to formalize knowledge that one concept comprises some other concepts and the latter for stating that one concept is more general than another concept. A concept itself describes an entity either from the real world, e.g., a car, or from nonmaterial descriptions, e.g., a sentence or a (physical) force.

Concepts form the basic building blocks of ontologies. In the context of this paper, concepts shall describe the basic entities of the input model. For example, in our running example depicted in Fig. 1 we use the concepts of Text, Sentence, Word, and Delimiter to construct an ontology for textual input. Concepts are also allowed to have attributes, i.e., properties that characterize concepts. For example, a person has a name, a birth day, and other specific properties that distinguishes one person from another. An attribute itself has a certain type like a string or a date. Note that for providing input models for test case generation, we only rely on enumeration types as basic type where someone has to specify the elements of the enumeration that corresponds to an attribute.

Definition 1

(Ontology). An ontology is a tuple \((C,A,D,\omega ,R, \tau , \psi )\) where C is a finite set of concepts, A is a finite set of attributes, D is a finite set of domain elements, \(\omega : C \mapsto 2^{A \times 2^D}\) is a function mapping concepts to a set of tuples specifying the attribute and its domain elements, and R is a finite set of tuples from \(C \times C \) stating that two concepts are related. The function \(\tau : R \times R \mapsto \{{{\mathtt{c}}},{{\mathtt{i}}}\}\) assigns a type to each relation using i for inheritance and c for composition. Furthermore, is a function mapping relationships solely of type c to its minimum and maximum arity. The arity is for specifying how many concepts a particular concept may comprise and ranges from 0 to any arbitrary natural number.

Note that this definition of ontologies assures that there is at the maximum one relation between two concepts. Hence, it is not possible to state that one concept is a sub-concept of another concept and that there is a compositional relationship between them as well. For simplicity, we further introduce the function \(dom: C \times A \mapsto 2^D\) that we will use later in this paper. The function dom returns the domain for an attribute of a given concept.

Obviously ontologies can also be represented in a graphical form. In Fig. 2, we depict the graphical representation of concepts and their relations using UML2 syntaxFootnote 1. Note that in the graphical representation the arity of the concept C1 of the composition relation is always 1. The minimum and maximum arity of concept C2 is given as follows: In case of * the minimum arity is 0 and the maximum arity is any value. Otherwise, we have a 1..* indicating a minimum arity of 1.

Fig. 2.
figure 2

Basic concepts, their attributes, and relationships considered for coming up with an ontology for a particular application domain.

Full size image

Example 1

Using the graphical representation of the text ontology from Fig. 1, we can easily obtain a formal description of the ontology accordingly to Definition 1. The text ontology comprises four concepts, i.e., \(C = \{{{\texttt {Text}}}, \) \({{\texttt {Sentence}}}, {\texttt {Word}}, {\texttt {Delimiter}}\}\), and one attribute leading to: \(A = \{{\texttt {id}}\}\). We further assume that we have three different words and only a period and a questionmark as delimiters. Hence, the domain comprises five elements, i.e.: .

Only delimiters and words have attributes. Hence, \(\omega \) can be defined as follows: \(\omega ({\texttt {Text}}) = \{\}\), \(\omega ({\texttt {Sentence}}) = \{\}\), , and . Finally, we only need to specify the relationships between concepts and the function \(\psi \). R can be easily obtained from Fig. 1: \(R = \{({\texttt {Text}}, {\texttt {Sentence}}), ({\texttt {Sentence}}, {\texttt {Delimiter}}), \) \(({\texttt {Sentence}}, {\texttt {Word}}) \}\). For these relations, we further specify their types: \(\tau ({\texttt {Text}}, {\texttt {Sentence}}) = {\texttt {c}}\), \(\tau ({\texttt {Sentence}}, {\texttt {Delimiter}}) = {\texttt {c}}\), and \(\tau ({\texttt {Sentence}}, {\texttt {Word}}) = {\texttt {c}}\). For \(\psi \) we have to take the numbers from the figure, where we assume * to be 2 leading to: \(\psi ({\texttt {Text}}, {\texttt {Sentence}}) = (1,2)\), \(\psi ({\texttt {Sentence}}, {\texttt {Delimiter}}) = (1,1)\), and \(\psi ({\texttt {Sentence}}, {\texttt {Word}}) = (1,2)\). It is worth noting that we can assume any arbitrary value of the arity for each relationship in case of *. However, it is not recommended to choose too high values, because - as we will see later - this increases the number of test cases to be generated.

When having a look at the graph representation of ontologies, we can obviously add structural means for characterizing ontologies. Concepts correspond to graph vertices, and relations to the edges of a graph. Considering the definition of ontologies, the relations are tuples comprising two concepts and a type and can be interpreted as directed edges. Hence, we can easily find a mapping from constraints to directed graphs and define the corresponding means for root vertices, leaf vertices, and cycles.

Definition 2

(Root concept; leaf concept). Given an ontology \((C,A,D,\omega , \) \(R,\psi )\). A concept \(c \in C\) is a root concept if and only if there exists no relation \((c',c,x) \in R\) for \(c'\in C\), \(x \in \{{{\mathtt{i}}},{{\mathtt{c}}}\}\). A concept \(c \in C\) is a leaf concept if and only if there is no relation \((c,c',x) \in R\) for \(c'\in C\), \({x \in \{{{\mathtt{i}}},{{\mathtt{c}}}\}}\).

Accordingly to this definition, there is one root vertex Text and two leaf vertices Delimiter and Word in the text ontology.

Definition 3

(Cyclic ontology). An ontology \((C,A,D,\omega ,R, \psi )\) is called a cyclic ontology if and only if (i) there is a relation \((c,c,x) \in R\) for a concept \(c\in C\) and \({x \in \{{{\mathtt{i}}},{{\mathtt{c}}}\}}\), or (ii) there are relations \((c_0,c_1,x_0), \ldots , (c_i,c_0,x_i)\) in R for \(i>0\), concepts \(c_0,\ldots ,c_i in C\) and \(x_0,\ldots ,x_i \in \{{\mathtt{i}},{\mathtt{c}}\}\). In this case the sequence of relations is called a cycle. If an ontology is not cyclic, it is called acyclic ontology.

Obviously the ontology for texts is an acyclic ontology. Note that for generating an input model for combinatorial testing, we rely on acyclic ontologies comprising exactly one root concept. We call such an ontology a well-formed ontology.

Definition 4

(Well-formed ontology). An ontology \((C,A,D,\omega ,R, \psi )\) is a well-formed ontology if and only if (i) it comprises exactly one root concept, (ii) it is acyclic, and (iii) where all its leaf concepts have attributes.

It is worth noting that rule (iii) in the definition of well-formed ontologies assures that we are able to construct test cases from ontologies.

In order to model the input for a particular application, we have to first come up with concepts that are used for describing that input. For each of these concepts, we further on define the attributes and finally the relations capturing the relationship between two concepts. For inheritance, we have to show that one concept is a generalization of the other or vice versa a specialization. For example, in the case of the automotive domain, we have the concepts of vehicle, car, and truck, where both truck and car are a specialization of the more general concept of vehicle. In case of composition, we have to ask whether one concept can be part of the description of another concept. In the text ontology, we know that a sentence has parts, i.e., its words, and also delimiters at the end. Hence, the concepts of word and delimiter have to be in compositional relation with sentence. As already mentioned, we also want to have exactly one root concept in the resulting ontology and no cycles.

3.2 Combinatorial Testing

Combinatorial testing (CT) is a method that aims to improve the effectiveness of software testing while lowering its cost at the same time. The essence of CT is that not all parameters contribute to failures but by interactions between relatively few parameters [6].

CT is rooted in the mathematical concept of combinatorics, which is used for the construction of combinatorial objects called Covering Arrays (CA). A fixed-value covering array denoted by \(CA(N, v^k, t)\) or CA(N, k, v, t) is an \(N \times k\) matrix of elements from a set of v symbols \(\{0,\ldots , (v - 1)\}\) such that every set of t columns contains each possible t-tuple of elements at least once where t is the strength of the covering array. A mixed-value covering array is an extension of fixed value CA where \(k = k_1 + \ldots + k_n\) meaning that for each column \(k_1\) (\(i=1\) to n) it has \(v_1\) (\(i=1\) to n)different elements, respectively [7].

Hence, the input model required for combinatorial testing has to have variables and their domains. In addition, there might be constraints that restrict the number of valid value combinations. Formally, we are able to define combinatorial testing input models as follows:

Definition 5

(Combinatorial testing input model). A combinatorial testing input model is a tuple (V, DOM, CONS) where V is a set of variables, DOM is a function mapping variables from V to a set of values, i.e., the variables’ domains, and CONS is a set of constraints over variables that have to be fulfilled for each test case.

Table 1. A 2-way test suite for platform configuration
Full size table

Testing with a test suite of t-way covering array is called t-way testing [7]. It requires that every combination of any t parameter values in the software must be tested at least once. When \(t = 1\), it is called the Each Choice (EC) combination strategy. It becomes Pairwise Testing (PW) when \(t = 2\). When \(t = n\), it is called the All Combination (AC) strategy meaning that every combination of all n parameter values must be tested at least once. The input model for CT includes the number of parameters, the value range for each parameter, and any constraints among these parameters. For example [9], assume that there is an application which needs to be run on different platforms consisting of five components (or parameters): OS (Windows XP, Apple OS X, Red Hat Enterprise Linux), browser (Internet Explorer, Firefox), protocol (IPv4, IPv6), CPU (Intel, AMD), and database (MySQL, Sybase, Oracle). There is a total of 72 (i.e., \(3\times 2 \times 2 \times 2 \times 3\)) possible platforms (or combinations). However, as shown in Table 1, it only requires 10 tests for conducting a 2-way or pairwise testing to cover all possible pairs of platform components.

According to Nie and Leung [16], CT has the following characteristics: (1) CT generates test cases by selecting values for parameters and by combining these values to form a CA. These test cases cover all t-way combinations of parameter values, where t is the strength specifying the number of parameters in combination; (2) CT uses a CA as the test suite. The CA aims to test as many parameter value combinations as possible to detect failures triggered by these interactions; (3) Not every parameter contributes to a fault, and some faults can only be triggered by a small number of parameters; (4) CT does not require knowledge about the implementation of SUT; and (5) Test generation for CT can be automated.

In addition, to conduct a CT-based test generation strategy, four aspects need to be considered [16]: (1) specifying the strength for CT as the corresponding CAs have different failure detection abilities with different cost. Two of the most widely used covering array generators are ACTS [26] and PICT [5]; (2) assign some specific test cases in advance; (3) considering constraints can increase the difficulty in applying CT; and (4) method to be used to generate test cases. In general, the methods can be categorized into four groups: greedy algorithm (e.g., AETG [2] and IPO [24]), heuristic search algorithm, mathematical method, and random method.

4 Ontology Conversion

Before introducing the ontology conversion formally, we first illustrate the underlying ideas using our text ontology.

Example 2

Let us explain our approach using the text ontology formally introduced in Example 1. First, have a look at the concept Sentence composed from Word and Delimiter. A test case capturing one particular Sentence can be considered as an instance or individual of Sentence. Each instance of Sentence, therefore, comprises 1 to n instances of Word, i.e., \(w_1,\ldots , w_n\), and one instance of Delimiter, i.e., d. Note that each instance of the leaf concepts Word and Delimiter comprises a particular value of their attributes, which can be obtained from their domains.

An instance of a concept that is not a leaf concept now comprise the instances of the concepts staying in relation. Hence, one instance of Sentence is \(w_1 \ldots w_n ~d\). This is exactly what we need for combinatorial testing. The \(w_i\)’s and d represent the variables of the input model and their domains come from leaf concepts, i.e., defined using the attributes and their domains that can be accessed using the dom function. For this example, instances of Word can be represented as a variable Word_id with a domain . Using the same idea for the delimiter concept, we finally obtain a combinatorial input model for Sentence comprising the variables Word_id_1, ..., Word_id_n, Delimiter_id, which represent the words \(w_1,\ldots ,w_n\) and the delimiter d, and their corresponding domains given when using the dom function.

In the following, we formally introduce the conversion of ontologies into an input model for combinatorial testing. Given an ontology \((C,A,D,\omega ,R, \tau , \psi )\), we describe the conversion for the three cases, i.e., concepts with attributes, inheritance relations, and compositional relations. For each case, we compute the combinatorial testing input model \(M^{CT}\) comprising its variables, their domains, and constraints denoted by \(V^{CT}\), \(DOM^{CT}\), and \(CONS^{CT}\) respectively, i.e., \(M^{CT} = (V^{CT},DOM^{CT}, CONS^{CT})\). Whenever needed we use a combinatorial testing algorithm CT(M, t) where M is a combinatorial testing input model and t the combinatorial strength, returning a test suite.

  • Concepts with attributes. Let us assume a concept \(c \in C\) with attributes \(a_1,\ldots ,a_n \in A\), \(n\ge 1\). This is the simplest case where we only have to construct input variables for combinatorial testing for each attribute of the concept. The domain of each combinatorial testing input variable is equivalent to the domain of its corresponding attribute, and there are no further constraints to add. I.e.: \(V^{CT} = \{ c\_a_1, \ldots , c\_a_n \}\), \(\forall _{i=1}^n: DOM^{CT}(c\_a_i) = dom(c,a_i)\), and \(CONS^{CT}= \{\}\).

  • Concepts and inheritance. Let us assume a concept \(c \in C\) having \(n \ge 1\) sub-concepts \(c_1,\ldots ,c_n \in C\) such that \(\forall _{i=1}^n:(c,c_i) \in R\). In this case, for c we have only one variable on side of combinatorial testing, and all its values come from the different values obtained when using the models from its sub-concepts and generating their combinatorial test suites. Let us assume \(M^{CT}_i\) be the combinatorial testing model of sub-concept \(c_i\), then \(M^{CT}\) of c is given as follows: \(V^{CT} = \{ c \}\), \(DOM^{CT}(c) = \bigcup _{i=1}^n CT(M^{CT}_i,t)\), and \(CONS^{CT} = \{\}\), where CT is an algorithm computing a combinatorial test suite of strength t.

  • Concepts and composition. Assume the case of two concepts \(c_1\) and \(c_2\) from C and a relation \((c_1,c_2) \in R\) of type \(\tau (c_1,c_2) = {\texttt {c}}\) in our ontology. This is the case we already discussed at the beginning of this section in Example 2. Depending on the minimum arity of the relation we have to consider two cases. If the minimum arity is zero, then there need not to be any instance of \(c_2\) attached to \(c_1\). In case the minimum arity is one, we have at least one individual of \(c_2\). We distinguish these two cases using a constraint.

    Before, we discuss how to handle the different instances of \(c_2\), i.e., \(c_2^1,\ldots ,c_2^m\). For a fixed m, we are able to construct a combinatorial testing input model. We only need to say that each instance is itself a variable and its domain is given from all values computed when using the combinatorial testing model of \(c_2\), i.e., \(M^{CT}(c_2)\). \(V^{CT} = \{ c_2^1,\ldots ,c_2^m \}\) and \(\forall _{i=1}^m: DOM^CT(c_2^i) = CT(M^{CT}(c_2),t) \cup \{ \epsilon \}\) where CT is an algorithm computing a combinatorial test suite of strength t.

    Note that \(\epsilon \) represents the empty value allowing a certain instance of \(c_2\) not to be relevant. We decided to use this, in order to finally come up with compositions of arbitrary length less than or equal to m. Because there might be the case that all instances of \(c_2\) have a value of \(\epsilon \), we have to consider two cases depending on the minimum arity of the relation. In case there need to be no instance of \(c_2\), i.e., when the minimum arity is zero, there is no need for a constraint, i.e., \(CONS^{CT} = \{\}\). Otherwise, \(CONS^{CT} = \{ \exists _i: 1 < i \le m \wedge c_2^i \not = \epsilon \}\) stating that at least one individual of \(c_2\) has to have a non empty value when computing the combinatorial test suite.

For general well-formed ontologies there might be cases where we have to combine the different rules described for computing a combinatorial test suite. For example, there might be a concept that has itself attributes and relations with other concepts. In such cases, we have to combine the variable, domains and constraints in order to compute the summary model of this concept for generating combinatorial test suites.

figure a

Algorithm 1 TC_GEN computes combinatorial test suites for the root concept of given ontology, and makes use of Algorithm 2 CT_ONT that recursively computes the combinatorial input models for the different concepts starting with the root node down to the leafs. In this algorithm the different cases are directly implemented. In the algorithm description we assume that the global variable m stores the number of instances that should be generated for compositional relations. The algorithm can be easily adapted in order to cope with a value of m to adapted for each concept. We only need to introduce a function that maps a concept to its maximum value of instances to be considered.

figure b

Obviously, CT_ONT terminates because we only consider finite and well-formed ontologies where there are no cycles and a finite number of concepts and relations. The algorithm traverses the whole graph and has therefore a computational complexity in the size of the graph when ignoring the computational complexity of combinatorial test suites during the traversal. The algorithm can be optimized to avoid multiple computations of input models, which may occur. The drawback of CT_ONT is that the number of computed test cases can be high. For example, let us consider the simplest case assuming a combinatorial strength of 2 and only 2 instances to be generated for every relation. If we consider the first relation with one leaf concept having one attribute with n elements in its domain, the number of generated tests is bound by n. If we use this result in a further step with 2 instances, we get \(n^2\) combinations because we have to combine all elements in the domain of the instances, which are n elements. In the next levels we obtain \(n^4\), \(n^{16}, \ldots , n^{2^m}\) test cases where m is the depth of the concept hierarchy. Hence, the depth of the ontology has to be small in order to be still feasible.

5 Case Study

For the case study, we use the Text ontology mentioned in previous sections for demonstration. In the following, we provide a step-by-step instruction for elaborating the process of applying the proposed algorithm TC_GEN to convert a given ontology to a CT-based input model.

Using the text ontology O, we invoke the function TC_GEN(O, t) at the beginning, which internally calls the function CT_ONT(Text, O, t), where Text is the root concept of our ontology O is (Algorithm 1, Line 2). In this step \(\omega (Text) = \{\}\) meaning that Text currently does not have any attributes. Therefore, the corresponding \(V^{CT}(Text)\) is empty and \(DOM^{CT}\) of Text is also empty (see Algorithm 2, Lines 1 to 6).

Since Text is not a leaf concept and there is one compositional relation between the concepts Text and Sentence, \(\tau (Text, Sentence)={\texttt {c}}\) with \(\psi (Text, \) \(Sentence)=(1,2)\). Then for the relation (Text, Sentence), we add two variables s1 and s2 to \(V^{CT}(Text)\). For each of these variables, \(DOM^{CT}(s\_i)\) is defined calling the combinatorial testing algorithm CT using the recursive call to CT_ONT, i.e., \(CT({\mathbf{CT\_ONT}}(Sentence, O, t), t) \cup \{\epsilon \}\) (Algorithm 2, Lines 18 to 20). Because \(\psi (Text, Sentence)=(1,2)\), we add the constraint \(s1 \not = \epsilon \vee s2 \not = \epsilon \) to \(CONS^{CT}\) for concept Text (Algorithm 2, Lines 21 to 23).

In order to determine the CT input model for Text, we first have to have the input model for concept Sentence. Obviously, \(DOM^{CT}(s1) = DOM^{CT}(s2)\) we obtain calling \(CT({\mathbf{CT\_ONT}}(Sentence, O, t), t) \cup \{\epsilon \}\). When invoking \(\mathbf {CT\_ONT}\) (Sentence, O, t), we see that Sentence is not a leaf concept because it has two compositional relations with \(\tau (Sentence, \) \(Delimiter) = {\texttt {c}} \), and \(\psi (Sentence, \) \(Delimiter)=(1,1)\) and \(\tau (Sentence, Word)= {\texttt {c}}\) and \(\psi (Sentence, Word)=(1,2)\), respectively. As a result, for (Sentence, Delimiter) we add one variable d to \(V^{CT}(Sentence)\) and let \(DOM^{CT}(d) = CT({\mathbf{CT\_ONT}}(Delimiter, O, t), t) \cup \{\epsilon \}\). For (Sentence, Word) we add two variables w1 and w2 to \(V^{CT}(Sentence)\) and let \(DOM^{CT}(wi) = CT({\mathbf{CT\_ONT}}(Word, O, k), k) \cup \{\epsilon \} \text { for } i=1,2\). In addition, we also add the two constraints \(d \not = \epsilon \) and \(w1 \not = \epsilon \vee w2 \not = \epsilon \) to \(CONS^{CT}\) for Sentence. For the recursive function call CT_ONT(Delimiter, O, k), and because Delimiter is a leaf concept we set \(V^{CT}(Delimiter)=\{id\}\), , and \(CONS^{CT}(Delimiter)=\{\}\), Therefore, the domain .

Regarding \(DOM^{CT}(wi) = CT({\mathbf{CT\_ONT}}(Word, O, t), t) \cup \{\epsilon \}\), let us again only focus on \(DOM^{CT}(w1) = CT({\mathbf{CT\_ONT}}(Word, O, t), t) \cup \{\epsilon \}\) in this example, because the same applies for computing \(DOM^{CT}(w2)\). For CT_ONT(Word, O, t), since Word is a leaf concept and , we obtain which is the same for \(DOM^{CT}(w2)\).

When taking this together, we are now able to come up with a CT input model \(M^{CT}\) for Sentence: \(V^{CT}(Sentence) = \{w1,w2,d\}\), with , , and \(CONS^{CT}(Sentence) = \{w1 \not = \epsilon \vee w2 \not = \epsilon , d \not = \epsilon \}\).

Now we use ACTS 3.1 and IPOG to generate a 2-way (i.e., t = 2) CT-based test suite for the instances of Sentence, i.e., s1 and s2. The hardware environment used to carry all experiments is a Dell XPS Laptop with 2.6 GHz, Intel Core i7, 16 GB memory using Windows 10. In Table 2 we see the obtained 15 test cases, which is far less compared to a total of 48 (\({=}4\times 4\times 3\)) possible test cases given the corresponding input model extracted from the ontology and assuming strength 2.

Table 2. 2-way test suite for the instances s1, and s2 of concept Sentence
Full size table

Note that the test suite in Table 2 forms the domain for variables s1 and s2 of the Word concept. Hence, we obtain the following CT input model \(M^{CT}\) for Word: having 15 items from the test suite of Sentence and additionally \(\epsilon \), and, \(CONS^{CT}(Text) = \{s1 \not = \epsilon \vee s2 \not = \epsilon \}\). When using ACTS 3.1 and IPOG, we obtain 255 test cases for the concept Text from this input model and combinatorial strength 2. Obviously, with the exception of one test case that would capture the case where both sentences are empty, this test suite comprises all possible elements. If we would not restrict the number of instances for sentence to 2, we would be able to obtain fewer tests if the strength is less than the number of sentences.

From this case study we see that the proposed algorithm TC_GEN allows for obtaining combinatorial test suites from specified ontologies. It is obvious that the proposed algorithm can be easily improved avoiding unnecessary recursive calls. For this purpose, we need only to store the outcome of combinatorial testing for one concept whenever available and re-use it during the process. It is also worth mentioning that in the algorithm and the explanations we assume a certain amount of instances to be generated. This information might be given for each concept in the ontology. In addition, the algorithm does not specify any order in the sequence of attributes or instances. For example, in Table 2 we stated the order w1, w2 followed by d. Hence, for coming up with concrete test cases, we have to make use of such an order to obtain meaningful information to be used to stimulate the system under test.

6 Conclusions

In this paper, we introduced an algorithm that is able to obtain a combinatorial test suite for a given ontology. We discussed the basic foundations and outlined the basic principles behind the algorithm. In addition, we elaborate on a case study showing that the algorithm can be applied delivering combinatorial test suites. Furthermore, we discussed additional assumptions used in the case study and potential improvements of the algorithm, e.g., for avoiding the re-computation of test input models in a recursive algorithm call.

To the best of our knowledge the proposed algorithm is the first one allowing to obtain combinatorial test suites from ontologies that can also be applied to UML class diagrams. Previous research has mainly focused on obtaining combinatorial test suites from use cases, activity or sequence diagrams. Improvements might consider generating a CT input model for the root concept of an ontology via collecting all variables from their related concepts instead of obtaining domain information from those concepts. In future research, we will investigate in this direction and also provide an empirical evaluation with the purpose of showing the practicability of the proposed approach for practical applications and comparing different versions of test suite generation using ontologies and combinatorial testing. In particular, we will focus on the automated and autonomous driving domain where practitioners and researchers are currently working on ontologies describing driving scenarios.