Elsevier

Journal of Systems and Software

Volume 142, August 2018, Pages 223-236
Journal of Systems and Software

Enable more frequent integration of software in industry projects

https://doi.org/10.1016/j.jss.2018.05.002Get rights and content

Highlights

  • A review of 74 research papers and four books related to continuous practices.

  • Results from 20 interviews on continuous integration impediments.

  • Twelve factors that affect how often developers deliver software to the mainline.

  • A model that identifies continuous integration impediments.

  • Results from validation that includes 46 individuals in five different companies.

Abstract

Based on interviews with 20 developers from two case study companies that develop large-scale software-intensive embedded systems, this paper presents twelve factors that affect how often developers commit software to the mainline. The twelve factors are grouped into four themes: “Activity planning and execution”, “System thinking”, “Speed” and “Confidence through test activities”. Based on the interview results and a literature study we present the EMFIS model, which allows companies to explicate a representation of the organization's current situation regarding continuous integration impediments, and visualizes what the organization must focus on in order to enable more frequent integration of software. The model is used to perform an assessment of the twelve factors, where the ratings from participants representing the developers are summarized separately from ratings from participants representing the enablers (responsible for processes, development tools, test environments etc.). The EMFIS model has been validated in workshops and interviews, which in total included 46 individuals in five case study companies. The model was well received during the validation, and was appreciated for its simplicity and its ability to show differences in rating between developers and enablers.

Introduction

A build system with the capacity to support frequent integration builds is often described as a prerequisite for continuous integration and related practices. For example, the importance of keeping the build fast is stated in Martin Fowler's popular article about continuous integration (Fowler, 2006). Paul Duvall talks about “build scalability”, which indicates “how capable your build system is of handling an increase in the amount of code” (Duvall, 2007). In the same way, Larman and Vodde (2010) describe “a slower build” as the main problem when scaling a continuous integration system.

Although many books tend to focus on the build system and other technological solutions, we believe that this is only one of several difficulties for the practitioners who struggle with large-scale continuous integration implementations in industry: continuous integration is simply not continuous (or even continual) without frequent integration of new software from the developers. This is a separate problem which is not necessarily solved merely by accelerated builds.

In our previous work we have shown that there is a gap between how continuous integration is described in literature by e.g. Fowler (2006) and how the practice is implemented in industry (Ståhl et al., 2017). We have also investigated which additional factors other than the build system play a role when applying continuous integration in industry (Mårtensson et al., 2017a). Based on metrics and interview results from a large-scale industry project, we presented the factors that according to the developers themselves affect how often they commit software to the mainline.

According to the results from our study, the developers will commit less frequently if the delivery process is time-consuming, if it's too complicated to commit or if there is no evident value in committing often to the mainline. Behind these three main themes, we also presented a range of sub-categories such as architecture, test activities and administration. But the question still remains: Which are the impediments that have to be overcome in order to fill the gap between the implementations of continuous integration in industry and how the practice is described in literature?

The topic of this paper is to answer the following research question: Which are the impediments that have to be overcome in order to enable software developers to commit more frequently, and could a model be defined that can be used as a representation of an organization's current situation regarding those impediments?

We will focus on continuous integration implementations for large-scale software-intensive embedded systems (software systems combined with electronical and mechanical systems). In previous work we have found multiple problems related to both scale (Ståhl et al., 2017) and proximity to hardware (Mårtensson et al., 2016). As we wish to find solutions viable in some of the most difficult cases, we have focused on these industry segments in this study (see Section 7.3 for further discussion of generalizability).

The contribution of this paper is two-fold. First, it presents a new model that can be used by practitioners in industry to visualize what their organization must focus on in order to enable more frequent integration of software. Second, the paper presents interview results from large-scale industry projects that can give researchers and practitioners an improved understanding of the main factors that affect how often developers commit software. In this paper we have combined and built upon our previous work (Mårtensson et al., 2017a, Mårtensson et al., 2017b, Mårtensson et al., 2017c) with validation extended to 46 individuals in five companies, and an extended analysis of applicability.

The remainder of this paper is organized as follows. In the next section we present the research method, including a description of the case study companies. This is followed by a study of related literature in Section 3. In Section 4, we present an analysis of the interview results. In Section 5 we present the EMFIS model, followed by a presentation of the validation of the model in Section 6. Threats to validity are discussed in Section 7. The paper is then concluded in Section 8.

Section snippets

Overview of the research method

The research study reported in this paper consists of four major parts:

  • A systematic literature review, to investigate whether solutions to the research question have been previously presented in literature (presented in Section 3).

  • Interviews on continuous integration impediments, to understand how the developers themselves describe the things that affect how often they commit their software to the mainline (presented in Section 4).

  • Development of the EMFIS model: a new model that makes it

Reviewing literature

A natural first step to answer the research question stated in Section 1 was to conduct a literature review, in order to look for solutions related to the research question in related work. The question driving the review was: “How are limitations, challenges or impediments related to continuous integration implementations for large-scale software-intensive embedded systems described in literature?”

Interviews on continuous integration impediments

As a complement to the literature study in Section 3, we conducted a series of interviews in order to understand how the developers themselves describe the things that affect how often they integrate their software to the mainline. We wanted to understand what needed to be changed so that the interviewed developers would commit their software more frequently.

The EMFIS model

In the studies of related work that is presented in Section 3, we found descriptions of a range of impediments related to continuous integration. However, all of the reviewed publications and books tend to focus on one or a few problem areas, and are leaving out areas that other authors consider to be the core issues. In response to this, we developed the EMFIS Model (Enable More Frequent Integration of Software) based on the analysis of interview results presented in Section 4.

Validation of the EMFIS model

The validation of the EMFIS model (described in Section 5) was conducted in two phases. In the first phase we wanted to reach out to five organizations in order to validate the model in different contexts. In the second phase, we wanted to extend the number of individuals involved in the validation, but also to compare different setups for the assessment workshops.

Threats to validity

This section discusses threats to construct validity, internal validity and external validity. We find threats to validity relevant to discuss with regards to the systematic literature review (presented in Section 3), the interviews on continuous integration impediments (presented in Section 4) and the validation interviews and workshops (presented in Section 6).

Conclusion

In this paper we have presented which impediments that have to be overcome in order to enable software developers to commit their software more frequently, and presented a model that makes it possible for companies to explicate a representation of the organization's current situation regarding those impediments.

In a systematic literature review including 74 research papers and four books we investigated how limitations, challenges or impediments related to continuous integration implementations

Acknowledgements

The authors would like to thank all the participating engineers for their insights, patience and willingness to share their experiences and data with us.

Torvald Mårtensson is Distinguished Engineer in System Integration and Test at Saab AB. He has a background of twelve years in the aeronautics industry and another eight years in the telecom industry. His work has primarily revolved around systems integration and system testing of large-scale software systems, and he has published several research papers on the subject at conferences and in journals. He received an MSc degree from Linköping University, Sweden.

References (51)

  • M. Fowler

    Continuous Integration

    (2006)
  • H. Grandy et al.

    Specification based testing of automotive human machine interfaces

  • J. Humble et al.

    Continuous Delivery

    (2011)
  • J. Humble

    The Case for Continuous Delivery

    (2014)
  • I. Jacobson et al.

    The Unified Software Development Process

    (1999)
  • B. Kitchenham

    Procedures for performing systematic reviews

    Keele UK Keele Univ.

    (2004)
  • E. Knauss

    Supporting continuous integration by code-churn based test selection

  • E. Knauss et al.

    Research preview: supporting requirements feedback flows in iterative system development

  • C. Larman et al.

    Practicies For Scaling Lean & Agile Development – Large, Multisite, and Offshore Product Development with Large-Scale Scrum

    (2010)
  • D. Leffingwell

    Agile Software Requirements

    (2011)
  • F. Lier et al.

    Towards automated execution and evaluation of simulated prototype HRI experiments

  • D. Liu

    An integrated development environment for microservices

  • A. Messina

    The database-is-the-service pattern for microservice architectures

  • M. Mossige et al.

    Test generation for robotized paint systems using constraint programming in a continuous integration environment

  • M. Mossige et al.

    Testing robotized paint system using constraint programming: an industrial case study

  • Cited by (9)

    • Adapting Behavior Driven Development (BDD) for large-scale software systems

      2021, Journal of Systems and Software
      Citation Excerpt :

      In our proposed process, test automation and product development can run in parallel, resulting in reducing time to deliver by 30%. Early feedback loops: Previous research in software integration projects has shown that rapid feedback loops are essential to identify, evaluate, and fix the problems with software artifacts cheaply and quickly (Mårtensson et al., 2018). As large-scale product development is an iterative process; therefore, quick feedback loops regarding requirements, architecture, and testing can help identify and resolve the issues quickly.

    • Efficient and effective exploratory testing of large-scale software systems

      2021, Journal of Systems and Software
      Citation Excerpt :

      The ExET model is used in a simple two-step process, in order to both involve the testers and the stakeholders for the test activities. A simplistic approach was purposely selected, based on our experiences and positive feedback received in the validation phases in our previous studies (Mårtensson et al., 2018, 2019b). The results from this part of the research study was a model (the ExET model) which can show companies what they should prioritize in order to optimize exploratory testing in their organization.

    • Challenges and Opportunities of DevOps in Cyber-Physical Production Systems Engineering

      2023, Proceedings - 2023 IEEE 6th International Conference on Industrial Cyber-Physical Systems, ICPS 2023
    • Agile Beyond Teams and Feedback Beyond Software in Automotive Systems

      2022, IEEE Transactions on Engineering Management
    • Supporting Domain Experts by using Model-Based Equivalence Class Partitioning for Efficient Test Data Generation

      2019, IEEE International Conference on Emerging Technologies and Factory Automation, ETFA
    View all citing articles on Scopus

    Torvald Mårtensson is Distinguished Engineer in System Integration and Test at Saab AB. He has a background of twelve years in the aeronautics industry and another eight years in the telecom industry. His work has primarily revolved around systems integration and system testing of large-scale software systems, and he has published several research papers on the subject at conferences and in journals. He received an MSc degree from Linköping University, Sweden.

    Daniel Ståhl works as Practice Area Lead Software and License Handling, subject matter expert and architect at Ericsson AB, where he is responsible for the corporate strategy on continuous integration, delivery and deployment. He has been working professionally in software development since 2007, and has spent most of that time implementing and researching continuous practices in organizations ranging from single teams to tens of thousands of engineers. He received an MSc degree from Linköping University, Sweden, and a PhD degree from the University of Groningen, The Netherlands.

    Jan Bosch is professor of software engineering and director of the software research center at Chalmers University Technology in Gothenburg, Sweden. Earlier, he has worked as Vice President Engineering Process at Intuit Inc and as head of the Software and Application Technologies Laboratory at Nokia Research Center, Finland. Before joining Nokia, he headed the software engineering research group at the University of Groningen, The Netherlands, where he holds a professorship in software engineering. He received a MSc degree from the University of Twente, The Netherlands, and a PhD degree from Lund University, Sweden.

    View full text