Enable more frequent integration of software in industry projects
Introduction
A build system with the capacity to support frequent integration builds is often described as a prerequisite for continuous integration and related practices. For example, the importance of keeping the build fast is stated in Martin Fowler's popular article about continuous integration (Fowler, 2006). Paul Duvall talks about “build scalability”, which indicates “how capable your build system is of handling an increase in the amount of code” (Duvall, 2007). In the same way, Larman and Vodde (2010) describe “a slower build” as the main problem when scaling a continuous integration system.
Although many books tend to focus on the build system and other technological solutions, we believe that this is only one of several difficulties for the practitioners who struggle with large-scale continuous integration implementations in industry: continuous integration is simply not continuous (or even continual) without frequent integration of new software from the developers. This is a separate problem which is not necessarily solved merely by accelerated builds.
In our previous work we have shown that there is a gap between how continuous integration is described in literature by e.g. Fowler (2006) and how the practice is implemented in industry (Ståhl et al., 2017). We have also investigated which additional factors other than the build system play a role when applying continuous integration in industry (Mårtensson et al., 2017a). Based on metrics and interview results from a large-scale industry project, we presented the factors that according to the developers themselves affect how often they commit software to the mainline.
According to the results from our study, the developers will commit less frequently if the delivery process is time-consuming, if it's too complicated to commit or if there is no evident value in committing often to the mainline. Behind these three main themes, we also presented a range of sub-categories such as architecture, test activities and administration. But the question still remains: Which are the impediments that have to be overcome in order to fill the gap between the implementations of continuous integration in industry and how the practice is described in literature?
The topic of this paper is to answer the following research question: Which are the impediments that have to be overcome in order to enable software developers to commit more frequently, and could a model be defined that can be used as a representation of an organization's current situation regarding those impediments?
We will focus on continuous integration implementations for large-scale software-intensive embedded systems (software systems combined with electronical and mechanical systems). In previous work we have found multiple problems related to both scale (Ståhl et al., 2017) and proximity to hardware (Mårtensson et al., 2016). As we wish to find solutions viable in some of the most difficult cases, we have focused on these industry segments in this study (see Section 7.3 for further discussion of generalizability).
The contribution of this paper is two-fold. First, it presents a new model that can be used by practitioners in industry to visualize what their organization must focus on in order to enable more frequent integration of software. Second, the paper presents interview results from large-scale industry projects that can give researchers and practitioners an improved understanding of the main factors that affect how often developers commit software. In this paper we have combined and built upon our previous work (Mårtensson et al., 2017a, Mårtensson et al., 2017b, Mårtensson et al., 2017c) with validation extended to 46 individuals in five companies, and an extended analysis of applicability.
The remainder of this paper is organized as follows. In the next section we present the research method, including a description of the case study companies. This is followed by a study of related literature in Section 3. In Section 4, we present an analysis of the interview results. In Section 5 we present the EMFIS model, followed by a presentation of the validation of the model in Section 6. Threats to validity are discussed in Section 7. The paper is then concluded in Section 8.
Section snippets
Overview of the research method
The research study reported in this paper consists of four major parts:
- •
A systematic literature review, to investigate whether solutions to the research question have been previously presented in literature (presented in Section 3).
- •
Interviews on continuous integration impediments, to understand how the developers themselves describe the things that affect how often they commit their software to the mainline (presented in Section 4).
- •
Development of the EMFIS model: a new model that makes it
Reviewing literature
A natural first step to answer the research question stated in Section 1 was to conduct a literature review, in order to look for solutions related to the research question in related work. The question driving the review was: “How are limitations, challenges or impediments related to continuous integration implementations for large-scale software-intensive embedded systems described in literature?”
Interviews on continuous integration impediments
As a complement to the literature study in Section 3, we conducted a series of interviews in order to understand how the developers themselves describe the things that affect how often they integrate their software to the mainline. We wanted to understand what needed to be changed so that the interviewed developers would commit their software more frequently.
The EMFIS model
In the studies of related work that is presented in Section 3, we found descriptions of a range of impediments related to continuous integration. However, all of the reviewed publications and books tend to focus on one or a few problem areas, and are leaving out areas that other authors consider to be the core issues. In response to this, we developed the EMFIS Model (Enable More Frequent Integration of Software) based on the analysis of interview results presented in Section 4.
Validation of the EMFIS model
The validation of the EMFIS model (described in Section 5) was conducted in two phases. In the first phase we wanted to reach out to five organizations in order to validate the model in different contexts. In the second phase, we wanted to extend the number of individuals involved in the validation, but also to compare different setups for the assessment workshops.
Threats to validity
This section discusses threats to construct validity, internal validity and external validity. We find threats to validity relevant to discuss with regards to the systematic literature review (presented in Section 3), the interviews on continuous integration impediments (presented in Section 4) and the validation interviews and workshops (presented in Section 6).
Conclusion
In this paper we have presented which impediments that have to be overcome in order to enable software developers to commit their software more frequently, and presented a model that makes it possible for companies to explicate a representation of the organization's current situation regarding those impediments.
In a systematic literature review including 74 research papers and four books we investigated how limitations, challenges or impediments related to continuous integration implementations
Acknowledgements
The authors would like to thank all the participating engineers for their insights, patience and willingness to share their experiences and data with us.
Torvald Mårtensson is Distinguished Engineer in System Integration and Test at Saab AB. He has a background of twelve years in the aeronautics industry and another eight years in the telecom industry. His work has primarily revolved around systems integration and system testing of large-scale software systems, and he has published several research papers on the subject at conferences and in journals. He received an MSc degree from Linköping University, Sweden.
References (51)
- et al.
Testing robot controllers using constraint programming and continuous integration
Sour. Doc. Inf. Software Technol.
(2015) Systematic scalability assessment for feature oriented multi-tenant services
J. Syst. Software
(2016)- et al.
Modeling continuous integration practice differences in industry software development
J. Syst. Software
(2014) - et al.
The continuity of continuous integration: correlations and consequences
J. Syst. Software
(2017) Extreme Programming Explained: Embrace Change
(2005)Improving testing in an enterprise SOA with an architecture-based approach
- et al.(1979)
Continuous evolution through software architecture evaluation: a case study
J. Software Maint. Evol.
(2006)Continuous Integration
(2007)Scaling agile methods to regulated environments: an industry case study
Continuous Integration
Specification based testing of automotive human machine interfaces
Continuous Delivery
The Case for Continuous Delivery
The Unified Software Development Process
Procedures for performing systematic reviews
Keele UK Keele Univ.
Supporting continuous integration by code-churn based test selection
Research preview: supporting requirements feedback flows in iterative system development
Practicies For Scaling Lean & Agile Development – Large, Multisite, and Offshore Product Development with Large-Scale Scrum
Agile Software Requirements
Towards automated execution and evaluation of simulated prototype HRI experiments
An integrated development environment for microservices
The database-is-the-service pattern for microservice architectures
Test generation for robotized paint systems using constraint programming in a continuous integration environment
Testing robotized paint system using constraint programming: an industrial case study
Cited by (9)
Adapting Behavior Driven Development (BDD) for large-scale software systems
2021, Journal of Systems and SoftwareCitation Excerpt :In our proposed process, test automation and product development can run in parallel, resulting in reducing time to deliver by 30%. Early feedback loops: Previous research in software integration projects has shown that rapid feedback loops are essential to identify, evaluate, and fix the problems with software artifacts cheaply and quickly (Mårtensson et al., 2018). As large-scale product development is an iterative process; therefore, quick feedback loops regarding requirements, architecture, and testing can help identify and resolve the issues quickly.
Efficient and effective exploratory testing of large-scale software systems
2021, Journal of Systems and SoftwareCitation Excerpt :The ExET model is used in a simple two-step process, in order to both involve the testers and the stakeholders for the test activities. A simplistic approach was purposely selected, based on our experiences and positive feedback received in the validation phases in our previous studies (Mårtensson et al., 2018, 2019b). The results from this part of the research study was a model (the ExET model) which can show companies what they should prioritize in order to optimize exploratory testing in their organization.
Challenges and Opportunities of DevOps in Cyber-Physical Production Systems Engineering
2023, Proceedings - 2023 IEEE 6th International Conference on Industrial Cyber-Physical Systems, ICPS 2023Agile Beyond Teams and Feedback Beyond Software in Automotive Systems
2022, IEEE Transactions on Engineering ManagementSupporting Domain Experts by using Model-Based Equivalence Class Partitioning for Efficient Test Data Generation
2019, IEEE International Conference on Emerging Technologies and Factory Automation, ETFA
Torvald Mårtensson is Distinguished Engineer in System Integration and Test at Saab AB. He has a background of twelve years in the aeronautics industry and another eight years in the telecom industry. His work has primarily revolved around systems integration and system testing of large-scale software systems, and he has published several research papers on the subject at conferences and in journals. He received an MSc degree from Linköping University, Sweden.
Daniel Ståhl works as Practice Area Lead Software and License Handling, subject matter expert and architect at Ericsson AB, where he is responsible for the corporate strategy on continuous integration, delivery and deployment. He has been working professionally in software development since 2007, and has spent most of that time implementing and researching continuous practices in organizations ranging from single teams to tens of thousands of engineers. He received an MSc degree from Linköping University, Sweden, and a PhD degree from the University of Groningen, The Netherlands.
Jan Bosch is professor of software engineering and director of the software research center at Chalmers University Technology in Gothenburg, Sweden. Earlier, he has worked as Vice President Engineering Process at Intuit Inc and as head of the Software and Application Technologies Laboratory at Nokia Research Center, Finland. Before joining Nokia, he headed the software engineering research group at the University of Groningen, The Netherlands, where he holds a professorship in software engineering. He received a MSc degree from the University of Twente, The Netherlands, and a PhD degree from Lund University, Sweden.