Reasons for bottlenecks in very large-scale system of systems development

https://doi.org/10.1016/j.infsof.2014.05.004Get rights and content

Abstract

Context

System of systems (SoS) is a set or arrangement of systems that results when independent and useful systems are to be incorporated into a larger system that delivers unique capabilities. Our investigation showed that the development life cycle (i.e. the activities transforming requirements into design, code, test cases, and releases) in SoS is more prone to bottlenecks in comparison to single systems.

Objective

The objective of the research is to identify reasons for bottlenecks in SoS, prioritize their significance according to their effect on bottlenecks, and compare them with respect to different roles and different perspectives, i.e. SoS view (concerned with integration of systems), and systems view (concerned with system development and delivery).

Method

The research method used is a case study at Ericsson AB.

Results

Results show that the most significant reasons for bottlenecks are related to requirements engineering. All the different roles agree on the significance of requirements related factors. However, there are also disagreements between the roles, in particular with respect to quality related reasons. Quality related hinders are primarily observed and highly prioritized by quality assurance responsibles. Furthermore, SoS view and system view perceive different hinders, and prioritize them differently.

Conclusion

We conclude that solutions for requirements engineering in SoS context are needed, quality awareness in the organization has to be achieved end to end, and views between SoS and system view need to be aligned to avoid sub optimization in improvements.

Introduction

System of systems development is considered a complex task [1]. Multiple definitions of system of systems exist (cf. [1], [2]). All agree on that a system in a system of systems shall be possible to be distributed alone. However, a set of systems also shall have the capability to operate together in a system of systems. Different strategies of how to develop the system of systems are characterized in [1], namely virtual, collaborative, acknowledged, and directed. The system of systems investigated in this study is directed. According to the definition in [1] a directed SoS is characterized by a well specified purpose, the development is centrally managed, and that “component systems maintain an ability to operate independently, but their normal operational mode is subordinated to the central managed purpose [1].” Given the centrally managed purpose, it is common that requirements may affect multiple systems, which require coordination. In contrast, in single stand-alone system development individual systems are developed and not built with the intention to operate in a compound of multiple other systems.

A vast amount of research has been done on system of systems in different disciplines. In particular military systems have been researched intensively [1], but not with a specific focus on software intensive system of systems. Research acknowledges that developing system of systems is a very complex task [1], [3], [4]. Already known reasons that complicate the development are the high number of parties and stakeholders involved, and the challenge of continuously integrating the systems [3], [4], [5].

In large-scale software intensive system of systems, a large amount of requirements are transformed (e.g. specified in detail, designed, coded, and tested). That is, they flow through the development life-cycle. It was found that the flow should be continuous and bottlenecks as well as inventory should be avoided (see [6], [7], [8]). In past research projects we provided solutions for analyzing the flow of requirements to identify bottlenecks [7]. We used those solutions to identify bottlenecks in single system development and SoS development, and observed more severe bottlenecks in SoS development (see Section 3.3). Bottlenecks have negative consequences, such as work overload [6], [9], delays, finding problems late in the process [10], requirements becoming obsolete [11], and creation of stress in the organization [12].

Given the severance of bottlenecks in complex SoS development, and the negative consequences associated with them, there is a need to identify the reasons for bottlenecks in SoS development. In order to identify the reasons we conducted a case study at Ericsson AB, the company develops single stand-alone systems and system of systems. In the case study we elicited reasons from two views, the system of systems view (responsible for technical oversight and making sure that individual systems can be integrated) and the system view (developing a system that is part of the system of systems in sets of projects). The elicited reasons were prioritized from both views. The following contributions are made by this study:

  • Identification of reasons for bottlenecks from the system of systems and system views in the flow of developing software requirements.

  • Prioritization of reasons from both views.

  • Comparative analysis with respect to different roles (e.g. testers, requirements engineers, and so forth).

  • Comparative analysis of the system of systems view and system view with respect to reasons identified and their priorities.

The practical relevance of the research is given as there is a need to understand the most critical reasons in order to prioritize process improvements addressing them.

The remainder of the paper is structured as follows: In Section 3 we provide a description of the SoS types identified in literature, and relate the types to the studied SoS. Furthermore, Section 3 reviews work on requirements flow and bottlenecks, followed by a comparison of bottlenecks between system of systems and single systems. Section 2 represents the related work, focusing on system of systems as well as existing studies on flow and bottleneck analysis in the software engineering context. Section 4 presents the research method (case study), describing the case context, units of analysis, data collection and data analysis. Furthermore, threats to validity are presented. Section 5 provides the results of the study. The results are described from the SoS view, and the system view. Both views are compared with each other. Section 6 discusses the results, and Section 7 concludes the paper.

Section snippets

Systems of systems

Although relatively new concept, there have been numerous publications in the area of SoS design and development. For example, Ncube [13] discussed key challenges related to interoperability, requirements engineering processes and management of emergent behavior. Ncube also stated initiatives that aim to provide strategic research agendas and road-maps for addressing some of these key challenges. Lewis et al. [4] identified a set of characteristics of SoS, proposed an SoS life-cycle, and

System of systems context

The system of systems term has been used in different disciplines (e.g. health [26] and military [1]).

The system of systems engineering guide of the department of defense [1] describes four types of system of systems, i.e. virtual, collaborative, acknowledged and directed. They differ with respect to the purpose, management and maintenance/change mechanism, which are summarized in Table 1. Based on these aspects, the studied SoS falls under directed SoS. Moreover, in practice there is usually

Research methodology

As a research methodology a single case study with multiple units of analysis (cf. [27]) has been used.

Results

The results are split based on the definition of the units of analysis (i.e. SoS level and system view). For each of the two views we provide an overview of the reasons, elaborate on their priorities, and identify the most significant differences between reasons with respect to their prioritization, followed by an analysis of agreement between practitioners conducting the prioritization.

Discussion

In the discussion, we highlight interesting observations and relate them to existing theories and literature. Based on the observations we provide implications and action points for researchers and practitioners.

Conclusion

We made four contributions in this paper. First, we have identified the reasons that make it challenging to achieve a continuous flow in SoS development from (a) system of systems view and (b) individual system view. These reasons were identified in workshops and after the workshops clustered under six different areas. We highlighted reasons that explicitly refer to the SoS, while other reasons are more general, but amplified due to the SoS context.

Secondly, we have reported the most

Acknowledgements

We would like to thank all the participants in the study who provided valuable input. Thanks also to Ronald Jabangwe for proof-reading the paper. This work has been partially funded by ELLIIT, The Linköping-Lund Initiative on IT and Mobile Communication, www.elliit.liu.se, and the BESQ + research project funded by the Knowledge Foundation (Sweden) (Grant: 20100311).

References (49)

  • D.J. Anderson

    Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results

    (2003)
  • P. Middleton

    Lean software development: two case studies

    Softw. Qual. J.

    (2001)
  • K. Petersen, C. Wohlin, D. Baca, The waterfall model in large-scale development – state of the art vs. industrial case...
  • T. Morgan, Lean Manufacturing Techniques Applied to Software Development, Master’s thesis, Master Thesis at...
  • C. Ncube, On the engineering of systems of systems: Key challenges for the requirements engineering community, in:...
  • D. Savio, P.C. Anitha, P.P. Iyer, Considerations for a requirements engineering process model for the development of...
  • J. Dahmann, K. Baldwin, Implications of systems of systems on system design and engineering, in: Proceedings of the 6th...
  • D. Hybertson, Using models and abstraction to extend and unify systems, in: Proceedings of the IEEE/SMC International...
  • D. DeLaurentis, A taxonomy-based perspective for systems of systems design methods, in: Proceedings of the 2005 IEEE...
  • S. Bellomo, J. Smith, Attributes of effective configuration management for systems of systems, in: Proceedings of the...
  • R. Rendon, Using a modular open systems approach in defense acquisitions: Implications for the contracting process, in:...
  • C. Larman

    Agile and Iterative Development: A Manager’s Guide

    (2004)
  • K. Beck et al.

    Extreme Programming Explained: Embrace Change

    (2005)
  • D. Cumbo et al.

    Benchmarking performance measurement and lean manufacturing in the rough mill

    Forest Prod. J.

    (2006)
  • Cited by (13)

    • Determining a core view of research quality in empirical software engineering

      2023, Computer Standards and Interfaces
      Citation Excerpt :

      Reliability: There is a potential threat related to the misinterpretation of the collected data. To mitigate such threat, both data analysis and interpretation procedures were designed in advance, based on the works of Kuzniarz & Angelis [36], and Petersen, Khurum & Angelis [39]. Mainly, we normalized the importance values, given the dimensions were not equally distributed across the four main areas.

    • FLOW-assisted value stream mapping in the early phases of large-scale software development

      2016, Journal of Systems and Software
      Citation Excerpt :

      The work by Staron and Meding (2011) acknowledges the underlying contribution of VSM but that was not the object of their study. While the earlier work on identification of bottlenecks by Petersen et al. (2014a) and Petersen et al. (2014b) uses the typical visualizations used in a value stream analysis, but it does not use the VSM as a framework to guide the improvement activity in the organization. Rother and Shook (1999) presented the guidelines to conduct VSM for manufacturing processes where both material and information flow are mapped.

    • Evaluation of simulation-assisted value stream mapping for software product development: Two industrial cases

      2015, Information and Software Technology
      Citation Excerpt :

      Continuous flow means that work items flow through the process at a continuous pace. A detailed account of definitions of continuous flow and reasons for the lack thereof is presented in [47,48]. A lack of continuous flow, in combination with the waiting times, is a first indication of waste in the process, and should be further investigated [8].

    • Temporal Complexity in Information Systems Development Flow: Challenges and Recommendations

      2024, Communications of the Association for Information Systems
    View all citing articles on Scopus
    View full text