research-article

Studying the impact of adopting continuous integration on the delivery time of pull requests

Authors:

João Helis Bernardo,

Daniel Alencar da Costa,

Uirá KuleszaAuthors Info & Claims

MSR '18: Proceedings of the 15th International Conference on Mining Software Repositories

Pages 131 - 141

https://doi.org/10.1145/3196398.3196421

Published: 28 May 2018 Publication History

Abstract

Continuous Integration (CI) is a software development practice that leads developers to integrate their work more frequently. Software projects have broadly adopted CI to ship new releases more frequently and to improve code integration. The adoption of CI is motivated by the allure of delivering new functionalities more quickly. However, there is little empirical evidence to support such a claim. Through the analysis of 162,653 pull requests (PRs) of 87 GitHub projects that are implemented in 5 different programming languages, we empirically investigate the impact of adopting CI on the time to deliver merged PRs. Surprisingly, only 51.3% of the projects deliver merged PRs more quickly after adopting CI. We also observe that the large increase of PR submissions after CI is a key reason as to why projects deliver PRs more slowly after adopting CI. To investigate the factors that are related to the time-to-delivery of merged PRs, we train regression models that obtain sound median R-squares of 0.64-0.67. Finally, a deeper analysis of our models indicates that, before the adoption of CI, the integration-load of the development team, i.e., the number of submitted PRs competing for being merged, is the most impactful metric on the time to deliver merged PRs before CI. Our models also reveal that PRs that are merged more recently in a release cycle experience a slower delivery time.

References

[1]

Kent Beck. 2000. Extreme Programming Explained: Embrace Change. Addison-Wesley Professional.

Digital Library

[2]

DJ Best and DE Roberts. 1975. Algorithm AS 89: the upper tail probabilities of Spearman's rho. Journal of the Royal Statistical Society. Series C (Applied Statistics) 24, 3 (1975), 377--379.

[3]

Jiyao Chen, Richard R Reilly, and Gary S Lynn. 2005. The impacts of speed-to-market on new product success: the moderating effects of uncertainty. IEEE Trans. Eng. Manage. 52, 2 (2005), 199--212.

[4]

Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, and Aditya Ghose. 2015. Predicting Delays in Software Projects Using Networked Classification (T). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 353--364.

Digital Library

[5]

Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, and Aditya Ghose. 2017. Predicting the delay of issues with due dates in software projects. Empirical Software Engineering Journal (2017), 1--41.

Digital Library

[6]

Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114, 3 (1993), 494.

[7]

Kevin Crowston, Hala Annabi, and James Howison. 2003. Defining open source software project success. ICIS 2003 Proceedings (2003), 28.

[8]

Daniel Alencar da Costa, Surafel Lemma Abebe, Shane McIntosh, Uirá Kulesza, and Ahmed E Hassan. 2014. An empirical study of delays in the integration of addressed issues. In Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on. IEEE, 281--290.

Digital Library

[9]

Daniel Alencar da Costa, Shane McIntosh, Uirá Kulesza, and Ahmed E. Hassan. 2016. The Impact of Switching to a Rapid Release Cycle on the Integration Delay of Addressed Issues: An Empirical Study of the Mozilla Firefox Project. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR '16). ACM, New York, NY, USA, 374--385.

Digital Library

[10]

Adam Debbiche, Mikael Dienér, and Richard Berntsson Svensson. 2014. Challenges when adopting continuous integration: A case study. In International Conference on Product-Focused Software Process Improvement. Springer, 17--32.

[11]

Paul Duvall, Stephen M Matyas, and Andrew Glover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series). Addison-Wesley Professional.

Digital Library

[12]

Martin Fowler and Matthew Foemmel. 2006. Continuous integration. ThoughtWorks) http://www.thoughtworks.com/ContinuousIntegration.pdf (2006), 122.

[13]

Emanuel Giger, Martin Pinzger, and Harald Gall. 2010. Predicting the fix time of bugs. In Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering. ACM, 52--56.

Digital Library

[14]

Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An exploratory study of the pull-based software development model. In Proceedings of the 36th International Conference on Software Engineering. 345--355.

Digital Library

[15]

Georgios Gousios and Diomidis Spinellis. 2012. GHTorrent: GitHub's data from a firehose. In Mining software repositories (msr), 2012 9th ieee working conference on. 12--21.

Digital Library

[16]

Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, and Arie Van Deursen. 2015. Work practices and challenges in pull-based development: the integrator's perspective. In Proceedings of the 37th International Conference on Software Engineering-Volume 1. 358--368.

Digital Library

[17]

Frank Harrell. 2015. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Springer.

[18]

Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering - ASE 2016.

Digital Library

[19]

Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2014. An Introduction to Statistical Learning: With Applications in R. Springer Publishing Company, Incorporated.

Digital Library

[20]

Yujuan Jiang, Bram Adams, and Daniel M German. 2013. Will my patch make it? and how fast? case study on the linux kernel. In Mining Software Repositories (MSR), 2013 10th IEEE Working Conference on. IEEE, 101--110.

Digital Library

[21]

Teemu Karvonen, Woubshet Behutiye, Markku Oivo, and Pasi Kuvaja. 2017. Systematic literature review on the impacts of agile release engineering practices. Information and Software Technology 86 (2017), 87 -- 100.

Digital Library

[22]

Eero Laukkanen, Maria Paasivaara, and Teemu Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration - A Case Study. In Proceedings of the 2015 Agile Conference (AGILE '15). IEEE Computer Society, 11--20.

Digital Library

[23]

Shane Mcintosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2016. An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. Empirical Softw. Engg. 21, 5 (2016), 2146--2189.

Digital Library

[24]

Mathias Meyer. 2014. Continuous integration and its tools. IEEE Softw. 31, 3 (2014), 14--16.

[25]

Nachiappan Nagappan and Thomas Ball. 2005. Use of relative code churn measures to predict system defect density. In Software Engineering, 2005. ICSE 2005. Proceedings. 27th International Conference on. IEEE, 284--292.

Digital Library

[26]

Dewayne E. Perry, Adam A. Porter, and Lawrence G. Votta. 2000. Empirical Studies of Software Engineering: A Roadmap. In Proceedings of the Conference on The Future of Software Engineering (ICSE '00). ACM, 345--355.

Digital Library

[27]

J. Romano, J.D. Kromrey, J. Coraggio, and J. Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen'sd for evaluating group differences on the NSSE and other surveys?. In annual meeting of the Florida Association of Institutional Research. 1--3.

[28]

Adrian Schroter, Adrian Schröter, Nicolas Bettenburg, and Rahul Premraj. 2010. Do stack traces help developers fix bugs?. In Mining Software Repositories (MSR), 2010 7th IEEE Working Conference on. IEEE, 118--121.

[29]

Ken Schwaber. 1997. SCRUM Development Process. In Business Object Design and Implementation, Dr Jeff Sutherland, Cory Casanave, Joaquin Miller, Dr Philip Patel, and Glenn Hollowell (Eds.). Springer London, 117--134.

[30]

Emad Shihab, Akinori Ihara, Yasutaka Kamei, Walid M Ibrahim, Masao Ohira, Bram Adams, Ahmed E Hassan, and Ken-ichi Matsumoto. 2010. Predicting reopened bugs: A case study on the eclipse project. In Reverse Engineering (WCRE), 2010 17th Working Conference on. IEEE, 249--258.

Digital Library

[31]

Daniel Ståhl and Jan Bosch. 2014. Modeling Continuous Integration Practice Differences in Industry Software Development. J. Syst. Softw. 87 (2014), 48--59.

Digital Library

[32]

Bogdan Vasilescu, Stef Van Schuylenburg, Jules Wulms, Alexander Serebrenik, and Mark GJ van den Brand. 2014. Continuous integration in a social-coding world: Empirical evidence from GitHub. In Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on. IEEE, 401--405.

Digital Library

[33]

Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and productivity outcomes relating to continuous integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2015.

Digital Library

[34]

Daniel S Wilks. 2011. Statistical methods in the atmospheric sciences. Vol. 100. Academic press.

[35]

David F. Williamson, Robert A. Parker, and Juliette S. Kendrick. 1989. The box plot: A simple visual method to interpret data. Annals of Internal Medicine 110 (1989), 916--921.

[36]

Krzysztof Wnuk, Tony Gorschek, and Showayb Zahda. 2013. Obsolete software requirements. Information and Software Technology 55, 6 (2013), 921--940.

Digital Library

[37]

Claes Wohlin, Min Xie, and Magnus Ahlgren. 1995. Reducing time to market through optimization with respect to soft factors. In The Engineering Management Conference. 116--121.

[38]

Yue Yu, Gang Yin, Tao Wang, Cheng Yang, and Huaimin Wang. 2016. Determinants of pull-based development in the context of continuous integration. Sci. China Inf. Sci. 59, 8 (2016).

[39]

Yangyang Zhao, Alexander Serebrenik, Yuming Zhou, Vladimir Filkov, and Bogdan Vasilescu. 2017. The impact of continuous integration on other software development practices: a large-scale empirical study. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 60--71.

Digital Library

Cited By

Bernardo JDa Costa DMedeiros SKulesza USpinellis DConstantinou EBacchelli A(2024)How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub ActionsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644915(665-676)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644915
Khatami AZaidman A(2024)State‐of‐the‐practice in quality assurance in Java‐based open source software developmentSoftware: Practice and Experience10.1002/spe.332154:8(1408-1446)Online publication date: 4-Mar-2024
https://doi.org/10.1002/spe.3321
Silva GBezerra CUchôa AMachado I(2023)What Factors Affect the Build Failures Correction Time? A Multi-Project StudyProceedings of the 17th Brazilian Symposium on Software Components, Architectures, and Reuse10.1145/3622748.3622753(41-50)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3622748.3622753
Show More Cited By

Recommendations

An exploratory study of the pull-based software development model
ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

The advent of distributed version control systems has led to the development of a new paradigm for distributed software development; instead of pushing changes to a central repository, developers pull them from other repositories and merge them ...
The impact of a continuous integration service on the delivery time of merged pull requests
Abstract
Continuous Integration (CI) is a software development practice that builds and tests software frequently (e.g., at every push). One main motivator to adopt CI is the potential to deliver software functionalities more quickly than not using CI. ...
Rugby: an agile process model based on continuous delivery
RCoSE 2014: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering

In this paper we introduce Rugby, an agile process model that includes workflows for the continuous delivery of software. It allows part-timers to work in a project-based organization with multiple projects for the rapid delivery of prototypes and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MSR '18: Proceedings of the 15th International Conference on Mining Software Repositories

May 2018

627 pages

ISBN:9781450357166

DOI:10.1145/3196398

General Chair:
Andy Zaidman
Delft University of Technology, Netherlands
,
Program Chairs:
Yasutaka Kamei
Kyushu University, Japan
,
Emily Hill
Drew University

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 28 - 29, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
395
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bernardo JDa Costa DMedeiros SKulesza USpinellis DConstantinou EBacchelli A(2024)How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub ActionsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644915(665-676)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644915
Khatami AZaidman A(2024)State‐of‐the‐practice in quality assurance in Java‐based open source software developmentSoftware: Practice and Experience10.1002/spe.332154:8(1408-1446)Online publication date: 4-Mar-2024
https://doi.org/10.1002/spe.3321
Silva GBezerra CUchôa AMachado I(2023)What Factors Affect the Build Failures Correction Time? A Multi-Project StudyProceedings of the 17th Brazilian Symposium on Software Components, Architectures, and Reuse10.1145/3622748.3622753(41-50)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3622748.3622753
Freitas GBernardo JSizíLio GCosta DKulesza U(2023)Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613403(1-10)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613403
Ouni ASaidani IAlomar EMkaouer M(2023)An Empirical Study on Continuous Integration Trends, Topics and Challenges in Stack OverflowProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593485(141-151)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593485
Saraiva DDa Costa DKulesza USizílio GNeto JCoelho RNagappan M(2023)Unveiling the Relationship Between Continuous Integration and Code Coverage2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00043(247-259)Online publication date: May-2023
https://doi.org/10.1109/MSR59073.2023.00043
Maipradit RWang DThongtanunam PKula RKamei YMcIntosh S(2023)Repeated Builds During Code Review: An Empirical Study of the OpenStack Community2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00030(153-165)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00030
Bernardo Jda Costa DKulesza UTreude C(2023)The impact of a continuous integration service on the delivery time of merged pull requestsEmpirical Software Engineering10.1007/s10664-023-10327-628:4Online publication date: 30-Jun-2023
https://dl.acm.org/doi/10.1007/s10664-023-10327-6
Rostami Mazrae PMens TGolzadeh MDecan A(2023)On the usage, co-usage and migration of CI/CD tools: A qualitative analysisEmpirical Software Engineering10.1007/s10664-022-10285-528:2Online publication date: 7-Mar-2023
https://dl.acm.org/doi/10.1007/s10664-022-10285-5
Zhang CChen BHu JPeng XZhao W(2022)BuildSonic: Detecting and Repairing Performance-Related Configuration Smells for Continuous Integration BuildsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556923(1-13)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556923
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten