Empirical research on concurrent software testing: A systematic mapping study

Abstract

Background: Concurrent software testing is a costly and difficult task, especially due to the exponential increase in the test sequences caused by non-determinism. Such an issue has motivated researchers to develop testing techniques that select a subset of the input domain that has a high probability of revealing faults. Academics and industrial practitioners rarely use most concurrent software testing techniques because of the lack of data about their applicability. Empirical evidence can provide an important scientific basis for the strengths and weaknesses of each technique to help researchers and practitioners choose concurrent testing techniques appropriate for their environments.

Aim: This paper gathers and synthesizes empirical research on concurrent software testing to characterize the field and the types of empirical studies performed.

Method: We performed a systematic mapping study to identify and analyze empirical research on concurrent software testing techniques. We provide a detailed analysis of the studies and their design choices.

Results: The primary findings are: (1) there is a general lack of empirical validation of concurrent software testing techniques, (2) the type of evaluation method varies with the type of technique, (3) there are some key challenges to empirical study design in concurrent software testing, and (4) there is a dearth of controlled experiments in concurrent software testing.

Conclusions: There is little empirical evidence available about some specific concurrent testing techniques like model-based testing and formal testing. Overall, researchers need to perform more empirical work, especially real-world case studies and controlled experiments, to validate properties of concurrent software testing techniques. In addition, researchers need to perform more analyses and synthesis of the existing evidence. This paper is a first step in that direction.

Introduction

The availability of multicore processors and inexpensive clusters has increased the demand both for concurrent applications and for testing techniques for their validation. Modern business applications use concurrency to improve overall system performance, consequently, researchers have developed a variety of testing techniques for concurrent software. However, testing teams generally rely on their own knowledge and experience when choosing technique(s) for each project, resulting in the repeated selection of same technique(s), whether or not they are most appropriate.

One of the difficulties in transfer of knowledge and research results from academia to industry is the lack of evidence of the applicability of results and techniques to specific software projects [1]. Such evidence should be gathered via empirical software engineering (ESE) methods that provide insights into the benefits and limits of each technique [2]. Secondary studies on concurrent software testing focus on the categorization of testing techniques, methodologies, and tools [3], [4], [5], [6] (see Section 2 for more details). However, empirical validation for concurrent software testing techniques is still lacking [1], [2].

Researchers use controlled experiments, case studies, and surveys as empirical methods to evaluate new techniques and new research. These methods produce data that helps researchers and practitioners decide whether a technique is appropriate for a given context. These empirical methods also help researchers identify specific factors that impact on the effectiveness of techniques and lead to empirically-based decisions about research and practice [7]. Therefore, they provide an important scientific basis for software engineering [8].

This paper discusses a systematic mapping study to understanding the current state-of-the-art of the empirical research on concurrent software testing research. The overall goal is to gather and synthesize empirical research and help developers evaluate the strength of evidence for the findings. The systematic mapping aims at:

• providing an overview of the empirical studies about concurrent software testing;

• identifying the concurrent software testing techniques that have empirical studies and the type of validation approach used;

• analyzing the strength of the empirical studies and discussing the findings and limitations of the evidence;

• discussing the design of the empirical studies along with the challenges and research opportunities;

• identifying gaps in empirical research on concurrent software testing; and

• providing guidance for the design of empirical studies about concurrent software testing.

The remainder of the paper is organized as follows. Section 2 provides an overview of the related work. Section 3 describes the systematic mapping protocol. Section 4 discusses the strengths of the empirical studies identified in the mapping study. Section 5 answers the research question trough the study results. Section 6 addresses the way researchers conduct empirical studies in for concurrent software testing and provides a guide for the planning of new studies. Section 7 discusses the limitations and validity threats of the mapping study. Finally, Section 8 provides the paper conclusions and suggests future work.