skip to main content
10.1145/3238147.3238171acmconferencesArticle/Chapter ViewAbstractPublication PagesaseConference Proceedingsconference-collections
research-article

Noise and heterogeneity in historical build data: an empirical study of Travis CI

Published: 03 September 2018 Publication History

Abstract

Automated builds, which may pass or fail, provide feedback to a development team about changes to the codebase. A passing build indicates that the change compiles cleanly and tests (continue to) pass. A failing (a.k.a., broken) build indicates that there are issues that require attention. Without a closer analysis of the nature of build outcome data, practitioners and researchers are likely to make two critical assumptions: (1) build results are not noisy; however, passing builds may contain failing or skipped jobs that are actively or passively ignored; and (2) builds are equal; however, builds vary in terms of the number of jobs and configurations.
To investigate the degree to which these assumptions about build breakage hold, we perform an empirical study of 3.7 million build jobs spanning 1,276 open source projects. We find that: (1) 12% of passing builds have an actively ignored failure; (2) 9% of builds have a misleading or incorrect outcome on average; and (3) at least 44% of the broken builds contain passing jobs, i.e., the breakage is local to a subset of build variants. Like other software archives, build data is noisy and complex. Analysis of build data requires nuance.

References

[1]
Bram Adams and Shane McIntosh. 2016. Modern Release Engineering in a Nutshell: Why Researchers should Care. In Proceedings of the International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 5. 78ś90.
[2]
Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software. In Proceedings of the International Conference on Software Analysis, Evolution, and Reengineering (SANER). 470ś481.
[3]
Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration. In Proceedings of the International Conference on Mining Software Repositories (MSR). 447ś450.
[4]
Panagiotis Dimitropoulos, Zeyar Aung, and Davor Svetinovic. 2017. Continuous integration build breakage rationale: Travis data case study. In International Conference on Infocom Technologies and Unmanned Systems (Trends and Future Directions) (ICTUS).
[5]
John Downs, Beryl Plimmer, and John G. Hosking. 2012. Ambient awareness of build status in collocated software teams. In Proceedings of the International Conference on Software Engineering (ICSE).
[6]
[7]
Ahmed E. Hassan and Ken Zhang. 2006. Using Decision Trees to Predict the Certiication Result of a Build. In Proceedings of the International Conference on Automated Software Engineering (ASE).
[8]
Foyzul Hassan, Shaikh Mostafa, Edmund S.L. Lam, and Xiaoyin Wang. 2017. Automatic Building of Java Projects in Software Repositories: A Study on Feasibility and Challenges. In Proceedings of the International Symposium on Empirical Software Engineering and Measurement (ESEM).
[9]
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-ofs in continuous integration: assurance, security, and lexibility. In Proceedings of Joint Meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE). 197ś207.
[10]
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and beneits of continuous integration in open-source projects. In Proceedings of the International Conference on Automated Software Engineering (ASE). 426ś437.
[11]
Peter Kampstra. 2008. Beanplot: A Boxplot Alternative for Visual Comparison of Distributions. Journal of Statistical Software 28, Code Snippet 1 (2008), 1ś9.
[12]
Noureddine Kerzazi, Foutse Khomh, and Bram Adams. 2014. Why Do Automated Builds Break? An Empirical Study. In Proceedings of the International Conference on Software Maintenance and Evolution (ICSME). 2014.26
[13]
Irwin Kwan, Adrian Schroter, and Daniela Damian. 2011. Does Socio-Technical Congruence Have an Efect on Software Build Success? A Study of Coordination in a Software Project. IEEE Transactions on Software Engineering (TSE) 37, 3 (2011), 307ś324.
[14]
Adriaan Labuschagne, Laura Inozemtseva, and Reid Holmes. 2017. Measuring the cost of regression testing in practice: a study of Java projects using continuous integration. In Proceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM Press, 821ś830.
[15]
Yang Luo, Yangyang Zhao, Wanwangying Ma, and Lin Chen. 2017. What are the Factors Impacting Build Breakage?. In Proceedings of the Web Information Systems and Applications Conference (WISA). IEEE.
[16]
Christian Macho, Shane McIntosh, and Martin Pinzger. 2018. Automatically Repairing Dependency-Related Build Breakage. In Proceedings of the International Conference on Software Analysis, Evolution, and Reengineering (SANER). 106ś117.
[17]
Shane McIntosh, Meiyappan Nagappan, Bram Adams, Audris Mockus, and Ahmed E. Hassan. 2015. A Large-Scale Empirical Study of the Relationship between Build Technology and Build Maintenance. Empirical Software Engineering 20, 6 (2015), 1587ś1633.
[18]
Audris Mockus. 2007. Software Support Tools and Experimental Work. In Empirical Software Engineering Issues. Critical Assessment and Future Directions: International Workshop, Dagstuhl Castle, Germany, June 26-30, 2006. Revised Papers, Victor R. Basili, Dieter Rombach, Kurt Schneider, Barbara Kitchenham, Dietmar Pfahl, and Richard W. Selby (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 91ś99.
[19]
Thomas Rausch, Waldemar Hummer, Philipp Leitner, and Stefan Schulte. 2017. An Empirical Analysis of Build Failures in the Continuous Integration Worklows of Java-Based Open-Source Software. In Proceedings of the International Conference on Mining Software Repositories (MSR). 345ś355.
[21]
Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum, Edward Aftandilian, and Robert Bowdidge. 2014. Programmers' build errors: a case study (at google). In Proceedings of the International Conference on Software Engineering (ICSE). 724ś734.
[22]
E. Burton Swanson. 1976. The Dimensions of Maintenance. In Proceedings of International Conference on Software Engineering (ICSE). 492ś497. http://dl.acm. org/citation.cfm?id=800253.807723
[23]
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. There and back again: Can you compile that snapshot? Journal of Software: Evolution and Process (JSEP) 29, 4 (2016), e1838.
[24]
Tijs van der Storm. 2008. Backtracking Incremental Continuous Integration. In Proceedings of the European Conference on Software Maintenance and Reengineering (CSMR).
[25]
Bogdan Vasilescu, Stef van Schuylenburg, Jules Wulms, Alexander Serebrenik, and Mark G. J. van den Brand. 2015. Continuous integration in a social-coding world: Empirical evidence from GitHub. **Updated version with corrections**. CoRR abs/1512.01862 (2015). arXiv: 1512.01862 http://arxiv.org/abs/1512.01862
[26]
Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE). 805ś816.
[27]
Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. 2018. Un-Break My Build: Assisting Developers with Build Repair Hints. In Proceedings of the International Conference on Program Comprehension (ICPC). To appear.
[28]
Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella. 2017.
[29]
A Tale of CI Build Failures: An Open Source and a Financial Organization Perspective. In Proceedings of the International Conference on Software Maintenance and Evolution (ICSME). 183ś193.
[30]
Timo Wolf, Adrian Schroter, Daniela Damian, and Thanh Nguyen. 2009. Predicting build failures using social network analysis on developer communication. In Proceedings of the International Conference on Software Engineering (ICSE).
[31]
Fiorella Zampetti, Simone Scalabrino, Rocco Oliveto, Gerardo Canfora, and Massimiliano Di Penta. 2017. How Open Source Projects Use Static Code Analysis Tools in Continuous Integration Pipelines. In Proceedings of the International Conference on Mining Software Repositories (MSR). 334ś344.

Cited By

View all
  • (2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
  • (2024)Dependency-Induced Waste in Continuous Integration: An Empirical Study of Unused Dependencies in the npm EcosystemProceedings of the ACM on Software Engineering10.1145/36608231:FSE(2632-2655)Online publication date: 12-Jul-2024
  • (2024)The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI BuildsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644924(543-555)Online publication date: 15-Apr-2024
  • Show More Cited By

Index Terms

  1. Noise and heterogeneity in historical build data: an empirical study of Travis CI

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
      September 2018
      955 pages
      ISBN:9781450359375
      DOI:10.1145/3238147
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 03 September 2018

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Automated Builds
      2. Build Breakage
      3. Continuous Integration

      Qualifiers

      • Research-article

      Conference

      ASE '18
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 82 of 337 submissions, 24%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)29
      • Downloads (Last 6 weeks)3
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
      • (2024)Dependency-Induced Waste in Continuous Integration: An Empirical Study of Unused Dependencies in the npm EcosystemProceedings of the ACM on Software Engineering10.1145/36608231:FSE(2632-2655)Online publication date: 12-Jul-2024
      • (2024)The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI BuildsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644924(543-555)Online publication date: 15-Apr-2024
      • (2024)Cost of Flaky Tests in Continuous Integration: An Industrial Case Study2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00037(329-340)Online publication date: 27-May-2024
      • (2024)On combining commit grouping and build skip prediction to reduce redundant continuous integration activityEmpirical Software Engineering10.1007/s10664-024-10477-129:6Online publication date: 30-Aug-2024
      • (2024)Enhancing continuous integration predictions: a hybrid LSTM-GRU deep learning framework with evolved DBSO algorithmComputing10.1007/s00607-024-01370-2107:1Online publication date: 26-Nov-2024
      • (2023)Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613403(1-10)Online publication date: 25-Sep-2023
      • (2023)An Empirical Approach to Enhance Performance for Scalable CORDIC-Based Deep Neural NetworksACM Transactions on Reconfigurable Technology and Systems10.1145/359622016:3(1-32)Online publication date: 20-Jun-2023
      • (2023)UniLoc: Unified Fault Localization of Continuous Integration FailuresACM Transactions on Software Engineering and Methodology10.1145/359379932:6(1-31)Online publication date: 8-May-2023
      • (2023)HybridCISave: A Combined Build and Test Selection Approach in Continuous IntegrationACM Transactions on Software Engineering and Methodology10.1145/357603832:4(1-39)Online publication date: 26-May-2023
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media