A language-independent approach to black-box testing using Erlang as test specification language

Highlights

• This paper presents a black-box approach to interface testing.

• The functional specification of a component's API is written using an abstract language and used to automatically generate and run tests.

• The methodology is illustrated using a real case study that reveals the potential of the technique.

• Several implementations in different programming languages were tested with a single test specification, and found several errors that had been missed in previous and more traditional testing activities.

• We create a reusable framework to test integration APIs of other components, the first stage in creating a complete testing framework.

Abstract

Integration of reused, well-designed components and subsystems is a common practice in software development. Hence, testing integration interfaces is a key activity, and a whole range of technical challenges arise from the complexity and versatility of such components.

In this paper, we present a methodology to fully test different implementations of a software component integration API. More precisely, we propose a black-box testing approach, based on the use of QuickCheck and inspired by the TTCN-3 test architecture, to specify and test the expected behavior of a component. We have used a real-world multimedia content management system as case study. This system offers the same integration API for different technologies: Java, Erlang and HTTP/XML. Using our method, we have tested all integration API implementations using the same test specification, increasing the confidence in its interoperability and reusability.

Introduction

Modular software systems are usually composed by integrated components. Software components integration and reuse is possible thanks to each component providing an interface, also called API (Application Programming Interface), which allows others to access its functionality. A complete usage protocol specification is needed to describe the API. This includes which operations the API provides, the types of the parameters of each operation and their return types, the order in which they must be executed, the allowed conditions under which they can be invoked, etc.

Testing this kind of API implementations is a very important task in the development process of complex systems. If the implementation of the specification is incorrect, components integration will fail causing system malfunctioning. In addition, both specifications and implementations evolve very frequently, and it is not uncommon that several implementations of a single API specification co-exist. This requires a suitable test approach that helps testers to test integration API implementations.

Unlike other stages of the software development cycle such as analysis, design, or implementation, in which practitioners use a set of well-known tools and methodologies, there is a lack of effective test strategies and technical facilitators that have made a significant impact in industry. Therefore, the main motivation for this work is the definition of a methodology that establishes the steps and technologies that can be used to improve, in particular, testing of integration API implementations.

Our proposal consists in using the API specification instead of the API implementation to generate, using a property-based testing approach (PBT: Derrick et al., 2010), meaningful test cases. PBT provides a powerful, high-level approach to testing: rather than focusing on individual test cases to encapsulate the behavior of a system, such behavior is specified using properties, expressed in a logical form (Derrick et al., 2010). This entails an important reduction of the lines of code in the corresponding test specification, i.e., the code manually written to test a system, compared to more traditional approaches, in which test cases are written by hand.

The proposed architecture is inspired by the Testing and Test Control Notation Version 3 (TTCN-3: Willcock et al., 2005, ETSI ES 201 873-1, 2018), which allows us to use the same test specification to test different implementations of the same API specification. However, unlike TTCN-3, we use properties, from which test cases are automatically generated, instead of writing the test suite manually. Properties are written using a functional programming language, specifically Erlang (Armstrong et al., 1996), and an existing property-based testing tool, QuickCheck (Claessen and Hughes, 2000, Arts et al., 2006) is used to generate and execute test cases.

The use of Erlang as test specification language is motivated by its capabilities to write concise and readable properties and models. Besides being a powerful, expressive, high-level programming language, Erlang integrates very well with other programming languages, such as C or Java. This integration allows for the use of Erlang as test specification language even if the System Under Test (SUT) is implemented in a different programming language. This helps us to achieve the goal of using a unique test specification language regardless of the programming language in which the SUT is implemented, avoiding the use of different languages to specify the system behavior to test. In addition, existing PBT libraries and tools available for Erlang, in particular QuickCheck (Claessen and Hughes, 2000, Arts et al., 2006), can be used to ease the task of defining test specifications, and to automatically generate and execute test cases. On the other hand, the TTCN-3 architecture is perfectly suited to our approach, because it splits the test code into an Abstract Test Suite (ATS) and some adapters that connect the abstract test suite with the SUT. Thus, with this approach, we are able to use the same test specification (in our case, properties), to test different implementations (our set of SUTs) of the same API specification.

We have used VoDKA Asset Manager, a content management component that provides an integration API for different programming languages (Java, HTTP/XML, and Erlang) as case study for our test strategy. This case study shows how the test approach is used with a real-world application, illustrating how all the implementations of the integration API are tested using the same test specification. In the process, we found and fixed several errors in our case study, previously overlooked by manual testing. We consider that the obtained results can be generalized to testing other integration API implementations.

As a result, we contribute a test methodology, especially oriented to test different implementations of an integration API: a black-box approach that uses properties to describe the API specification in an abstract way. This methodology has two strong advantages:

• The use of abstract properties avoids dependencies with the API implementations, minimizing the effort to test their functional requirements, and the functional equivalence between them.

• The maintenance of the test specification (i.e., the properties that describe the SUT) is significantly simplified. Apart from writing only one test specification per API specification (instead of one for each API implementation to test), not all kinds of changes in the SUT implementation imply changes in the test specification. Thus, we update the properties only if the API specification changes.

We present an implementation of this methodology using QuickCheck, which allows us to automatically generate and run specific test cases from the abstract properties. This is the first step in the creation of a reusable framework to test integration APIs, and we illustrate the process on the light of our case study.

The rest of the paper is organized as follows. Section 2 presents our case study, describes the problem, and analyzes different approaches which could be used for the same purpose, such as TTCN-3. Section 3 explains our proposal in detail. Section 4 summarizes the results and limitations of the approach. Finally, in Section 5, conclusions and future lines related to this work are included.