Assessing a requirements evolution approach: Empirical studies in the air traffic management domain

Highlights

• We conducted empirical studies in the ATM domain to evaluate the effectiveness of an approach to model requirements change.

• The studies involved participants with different level of knowledge in the approach and in the ATM domain.

• The studies show that the approach is effective in modeling requirements evolution.

• The studies show that domain knowledge and method knowledge have no observable effect on the effectiveness of the approach.

Abstract

Requirements evolution is still a challenging problem in engineering practices. In this paper, we report the results of the empirical evaluation of a novel approach for modeling and reasoning on evolving requirements. We evaluated the effectiveness of the approach in modeling requirements evolution by means of a series of empirical studies in the air traffic management (ATM) domain. As we also wanted to assess whether the knowledge of the method and/or the application domain influences the effectiveness of the approach, the studies involved researchers, master students and domain experts with different level of knowledge of the approach and of the ATM domain. The participants have applied the approach to a real evolutionary scenario which focuses on the introduction of a new queue management tool, the Arrival MANager (AMAN) and a new network for information sharing (SWIM) connecting the main ATM actors. The results from the studies show that the modeling approach is effective in capturing requirements evolution. In addition, domain knowledge and method knowledge do not have an observable effect on the effectiveness of the approach. Furthermore, the evaluation provided us useful insights on how to improve the modeling approach.

Introduction

The evolution of mission-critical requirements at enterprise level is known to be possible, but it is unknown whether it would happen: the known unknowns (Tran and Massacci, 2011). Unfortunately, large organizations cannot wait until the unknowns become known. The process of tendering and organizational restructuring requires a significant amount of time and planning. Decision makers at high-level must essentially bet on the final organizational solution and possibly minimize the risks that the solution turns out to be wrong. There is, thus, the need of approaches for evolving requirements that should help decision makers to select an optimal system design alternative that is resilient to requirements evolution.

In this paper we present the results of an empirical evaluation conducted on a requirements engineering approach to model and reason on requirements evolution (previously proposed in Tran and Massacci (2011)). The evaluation aimed to assess the effectiveness of the approach in modeling requirements evolution and whether the effectiveness depends on the analyst's level of knowledge of the approach and of the application domain. To this end, we conducted three empirical studies with participants having different level of knowledge of the modeling approach and of the application domain. Fig. 1 summarizes how our empirical studies developed along a two-year horizon. First, we have conducted a study within the research team who have proposed the approach to model evolving requirements. Then, we have pushed the envelope further by carrying out a series of workshops with domain experts and industry practitioners as in Ncube et al. (2007). Last, we conducted a study with MSc students.

As context for our evaluation, we have chosen the air traffic management (ATM) domain for three main reasons. First, ATM systems are complex and critical systems that are going through significant architectural, organizational, and operational changes as planned by the EU Single European Sky ATM Research (SESAR) Initiative (EUROCONTROL, 2003). Second, change management is a critical issue in the ATM domain. The need of system engineering techniques to support change management is well recognized (Graham et al., 2009). Last but not least there is a significant body of research about empirical evaluations of requirements engineering approaches in the ATM domain (Maiden and Robertson, 2005, Maiden et al., 2004, Ncube et al., 2007). For example, in Maiden et al. (2004), DMAN (Departure MANager), a system for managing departure of aircrafts, is used as context of evaluation. This makes it easier to benchmark our studies.

In our empirical evaluation, we have focused on changes associated with the introduction of a new decision supporting tool (the AMAN – Arrival MANager) and SWIM (System Wide Information Management) in the ATM domain.

The results from the studies show that the modeling approach is effective in capturing requirements evolution. In fact, the studies showed that it is possible for people different than the method's own inventor (such as students or domain experts) to build significantly large models, and identify possible ways for these models to evolve. Moreover, the studies have shown that domain knowledge and method knowledge have no observable effect on the effectiveness of the approach.

The paper is structured as follows. Section 2 introduces the context of our studies. Section 3 gives an overview of the approach to model requirements evolution being validated. We describe the research methodology in Section 4. Section 5 presents the analysis of the data collected during the studies. Section 6 summarizes the main findings. Section 7 discusses the threats to validity. Section 8 gives an overview of related works while Section 9 concludes the paper with the lessons learned.