Assessing a requirements evolution approach: Empirical studies in the air traffic management domain
Introduction
The evolution of mission-critical requirements at enterprise level is known to be possible, but it is unknown whether it would happen: the known unknowns (Tran and Massacci, 2011). Unfortunately, large organizations cannot wait until the unknowns become known. The process of tendering and organizational restructuring requires a significant amount of time and planning. Decision makers at high-level must essentially bet on the final organizational solution and possibly minimize the risks that the solution turns out to be wrong. There is, thus, the need of approaches for evolving requirements that should help decision makers to select an optimal system design alternative that is resilient to requirements evolution.
In this paper we present the results of an empirical evaluation conducted on a requirements engineering approach to model and reason on requirements evolution (previously proposed in Tran and Massacci (2011)). The evaluation aimed to assess the effectiveness of the approach in modeling requirements evolution and whether the effectiveness depends on the analyst's level of knowledge of the approach and of the application domain. To this end, we conducted three empirical studies with participants having different level of knowledge of the modeling approach and of the application domain. Fig. 1 summarizes how our empirical studies developed along a two-year horizon. First, we have conducted a study within the research team who have proposed the approach to model evolving requirements. Then, we have pushed the envelope further by carrying out a series of workshops with domain experts and industry practitioners as in Ncube et al. (2007). Last, we conducted a study with MSc students.
As context for our evaluation, we have chosen the air traffic management (ATM) domain for three main reasons. First, ATM systems are complex and critical systems that are going through significant architectural, organizational, and operational changes as planned by the EU Single European Sky ATM Research (SESAR) Initiative (EUROCONTROL, 2003). Second, change management is a critical issue in the ATM domain. The need of system engineering techniques to support change management is well recognized (Graham et al., 2009). Last but not least there is a significant body of research about empirical evaluations of requirements engineering approaches in the ATM domain (Maiden and Robertson, 2005, Maiden et al., 2004, Ncube et al., 2007). For example, in Maiden et al. (2004), DMAN (Departure MANager), a system for managing departure of aircrafts, is used as context of evaluation. This makes it easier to benchmark our studies.
In our empirical evaluation, we have focused on changes associated with the introduction of a new decision supporting tool (the AMAN – Arrival MANager) and SWIM (System Wide Information Management) in the ATM domain.
The results from the studies show that the modeling approach is effective in capturing requirements evolution. In fact, the studies showed that it is possible for people different than the method's own inventor (such as students or domain experts) to build significantly large models, and identify possible ways for these models to evolve. Moreover, the studies have shown that domain knowledge and method knowledge have no observable effect on the effectiveness of the approach.
The paper is structured as follows. Section 2 introduces the context of our studies. Section 3 gives an overview of the approach to model requirements evolution being validated. We describe the research methodology in Section 4. Section 5 presents the analysis of the data collected during the studies. Section 6 summarizes the main findings. Section 7 discusses the threats to validity. Section 8 gives an overview of related works while Section 9 concludes the paper with the lessons learned.
Section snippets
Application scenarios
The context of our study is the evolution in air traffic management procedures planned by the SESAR (Single European Sky ATM Research) programme which is building the future European air traffic management system. The application scenarios (Table 1 provides a list of technical documents) were provided by Deep Blue Srl, an Italian consultancy company specialized in human factors, safety and validation of ATM concepts and systems, which actively participates to the SESAR Initiative. The scenarios
The validated approach
This section gives an overview of an approach (Tran and Massacci, 2011) to deal with requirements evolution at design time. The ultimate objective of the approach is to help decision makers to select an optimal design solution so that the deployed system could be operational without (or with less) modification, while still keeping the development cost in budget. In this study, we aim to validate only the modeling aspect. Therefore, we do not discuss the reasoning part of the approach.
Research method
In this section we present our research questions and hypotheses, and the protocol followed to conduct our studies (Section 4.2).
Quantitative data analysis
We collected the artifacts produced by researchers, domain experts and students as summarized in Table 3. The table reports for researchers, domain experts and students the mean and standard deviations of size of baseline, size of changes, and number of branches for controllable and observable rules.
To take into account the quality of the evolution rules and requirements models generated by students and researchers, we asked to a Deep Blue consultant who was expert in the ATM domain to assess
Discussion
This section summarizes the main findings from the studies we conducted (see Table 6).
Threats to validity
We discuss the four main types of threats to validity (Wohlin et al., 2012) in what follows.
Related work
We discuss here the works reporting empirical studies that are related to requirements evolution and that have been conducted in the ATM domain.
A closely related work to ours is from Villela et al. (2010) who reported on a quasi-experiment conducted in the Ambient Assisted Living domain to assess the adequacy and feasibility of PLEvo-Scoping method (Villela et al., 2008). That method allows to identify and prioritize likely future adaptation needs and to select solutions to deal with them. The
Conclusions and lessons learnt
In this paper we reported the results of three studies that we have conducted in the ATM domain to evaluate the effectiveness of an approach to model requirements evolution and the impact that domain knowledge and method knowledge have on effectiveness.
The main findings from the studies are that the approach is effective in modeling requirements evolution. In fact, the studies showed that researchers, practitioners and students were able to produce significantly big requirements models, and
Acknowledgements
This work is partly supported by EU-FP7-IST-NoE-NESSOS, EU-EIT-ICTLabs, EU-FP7-SEC-SECONOMICS, PAT-TRISE.
Fabio Massacci received a MEng in 1993 and PhD in Computer Science andEngineering at University of Rome La Sapienza in 1998. He visited Cambridge University in 1996–97 and was visiting researcher at IRIT Toulouse in 2000. He joined the University of Siena as Assistant Professor in 1999, and in 2001 he went to Trento where he is now full professor. His research interests are in malware analysis, security economics, empirical validation of risk and security Requirements Methodologies, and
References (20)
- et al.
The TAME project: towards improvement-oriented software environments
IEEE Transactions on Software Engineering
(1988) - EUROCONTROL, 2003. ATM Strategy for the Years...
- EUROCONTROL, 2010. European Operational Concept Validation Methodology, EOCVM Version...
- et al.
Performance framework and influence model in ATM
- et al.
Specifying changes only – a case study on delta requirements
- et al.
Model-Driven Risk Analysis: The CORAS Approach
(2011) - et al.
Integrating creativity into requirements processes: experiences with an air traffic management system
- et al.
Model-driven requirements engineering: synchronising models in an air traffic management case study
- et al.
Security requirements engineering: the SI* modeling language and the secure tropos methodology
- et al.
Software requirements change taxonomy: evaluation by case study
Cited by (8)
Operational effectiveness evaluation of ground service for transit flight
2019, Xi Tong Gong Cheng Yu Dian Zi Ji Shu/Systems Engineering and ElectronicsA critical view over iStar visual constructs
2019, CEUR Workshop ProceedingsSecurity Requirements Elicitation from Airline Turnaround Processes
2018, Business and Information Systems EngineeringFramework for the elicitation and evolution of non-functional requirements based on knowledge management
2018, Avances en Ingenieria de Software a Nivel Iberoamericano, CIbSE 2018Assessment of aviation security risk management for airline turnaround processes
2017, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Requirements change management based on object-oriented software engineering with unified modeling language
2016, Proceedings of the IEEE International Conference on Software Engineering and Service Sciences, ICSESS
Fabio Massacci received a MEng in 1993 and PhD in Computer Science andEngineering at University of Rome La Sapienza in 1998. He visited Cambridge University in 1996–97 and was visiting researcher at IRIT Toulouse in 2000. He joined the University of Siena as Assistant Professor in 1999, and in 2001 he went to Trento where he is now full professor. His research interests are in malware analysis, security economics, empirical validation of risk and security Requirements Methodologies, and predictive models for vulnerabilities. With W. Joosen he co-founded the ESSOS, Engineering Secure Software and Systems Symposium which aims at bringing together Requirements and Software Engineers and Security experts. He is currently leading the Empirical Security Requirements and Risk Engineering Challenge (ERISE). He has been scientific coordinator of multimillion euro EU projects on security compliance, security engineering and secure evolution.
Federica Paci is a post doctoral researcher in the Security Group at the Department of Information Engineering and Computer Science of the University of Trento. She received her PhD in Information Technology from the University of Milan in February 2008. Dr. Paci has been a post-doctoral fellow at Purdue University from February 2008 to March 2009 under the supervision of Prof. Elisa Bertino. Her recent research interests focus on empirical evaluations of security engineering methods, security requirements evolution, and privacy management for Future Internet applications. Dr. Paci has published more that 40 contributions as papers in international conferences and journals, and chapter in international books. She has served as member of the program committee of various international conferences.
Le Minh Sang Tran is a PhD student at the ICT Doctoral School, University of Trento. She received a double-degree of Master in computer science from RWTH Aachen University and University of Trento in 2009. Her research interests include security engineering, empirical software engineering, software evolution, in particular requirements evolution which is also the focus of her dissertation under the supervision of Prof. Fabio Massacci.
Alessandra Tedeschi holds a PhD in Applied Mathematics from the University of L’Aquila and Rome La Sapienza. Her research interests includes the analysis and modeling of Complex Systems with Game Theory techniques, Safety and Security assessment in socio-technical domains, conflict detection and resolution algorithms for automated aviation systems. She is co-author of several scientific papers in the field of Statistical Mechanics, Software Engineering and Security and Dependability. She has been working with Deep Blue since 2007, where she has been involved as validation expert in EU and SESAR funded research projects.