skip to main content
10.1145/2950290.2950358acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Build system with lazy retrieval for Java projects

Published: 01 November 2016 Publication History

Abstract

In the modern-day development, projects use Continuous Integration Services (CISs) to execute the build for every change in the source code. To ensure that the project remains correct and deployable, a CIS performs a clean build each time. In a clean environment, a build system needs to retrieve the project's dependencies (e.g., guava.jar). The retrieval, however, can be costly due to dependency bloat: despite a project using only a few files from each library, the existing build systems still eagerly retrieve all the libraries at the beginning of the build.
This paper presents a novel build system, Molly, which lazily retrieves parts of libraries (i.e., files) that are needed during the execution of a build target. For example, the compilation target needs only public interfaces of classes within the libraries and the test target needs only implementation of the classes that are being invoked by the tests. Additionally, Molly generates a transfer script that retrieves parts of libraries based on prior builds. Molly's design requires that we ignore the boundaries set by the library developers and look at the files within the libraries. We implemented Molly for Java and evaluated it on 17 popular open-source projects. We show that test targets (on average) depend on only 9.97% of files in libraries. A variant of Molly speeds up retrieval by 44.28%. Furthermore, the scripts generated by Molly retrieve dependencies, on average, 93.81% faster than the Maven build system.

References

[1]
How not to download the Internet. http://blog. sonatype.com/2011/04/ how-not-to-download-the-internet.
[2]
WAD home page. https://github.com/Fingertips/ WAD/.
[3]
Your Maven build is slow. Speed it up! http:// zeroturnaround.com/rebellabs/ your-maven-build-is-slow-speed-it-up/.
[4]
B. Adams, K. De Schutter, H. Tromp, and W. De Meuter. The evolution of the Linux build system. Electronic Communications of the ECEASST, 8:1–16, 2008.
[5]
E. Aftandilian, R. Sauciuc, S. Priya, and S. Krishnan. Building useful program analysis tools using an extensible Java compiler. In International Working Conference on Source Code Analysis and Manipulation, pages 14–23, 2012.
[6]
Apache Camel. https://github.com/apache/camel.
[7]
Apache Commons Configuration. https://github.com/ apache/commons-configuration.
[8]
Apache Ivy. http://ant.apache.org/ivy.
[9]
Apache Maven. https://maven.apache.org.
[10]
Apache Maven Central Repository. http://search. maven.org.
[11]
ASM home page. http://asm.ow2.org/.
[12]
Bazel home page. http://bazel.io.
[13]
K. Beck. Extreme programming explained: embrace change. Addison-Wesley Professional, 2000.
[14]
K. Beck, M. Beedle, A. Van Bennekum, A. Cockburn, W. Cunningham, M. Fowler, J. Grenning, J. Highsmith, A. Hunt, R. Jeffries, et al. Manifesto for agile software development. 2001.
[15]
J. Bell and G. E. Kaiser. Unit test virtualization with VMVM. In International Conference on Software Engineering, pages 550–561, 2014.
[16]
J. Bell, E. Melski, G. Kaiser, and M. Dattatreya. Accelerating Maven by delaying test dependencies. In International Workshop on Release Engineering, pages 28–28, 2015.
[17]
E. Bodden, A. Sewe, J. Sinschek, H. Oueslati, and M. Mezini. Taming reflection: Aiding static analysis in the presence of reflection and custom class loaders. In International Conference on Software Engineering, pages 241–250, 2011.
[18]
T. Bodhuin, M. Di Penta, and L. Troiano. A search-based approach for dynamically re-packaging downloadable applications. In Conference of the Center for Advanced Studies on Collaborative Research, pages 27–41, 2007.
[19]
G. Booch. Object Oriented Design: With Applications. Benjamin/Cummings Pub., 1991.
[20]
Build in the Cloud: How the build system works. http://google-engtools.blogspot.com/2011/08/ build-in-cloud-how-build-system-works.html.
[21]
M. Christakis, K. R. M. Leino, and W. Schulte. Formalizing and verifying a modern build language. In International Symposium on Formal Methods, pages 643–657, 2014.
[22]
B. Dagenais and L. Hendren. Enabling static analysis for partial Java programs. In Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 313–328, 2008.
[23]
S. Elbaum, G. Rothermel, and J. Penix. Techniques for improving regression testing in continuous integration development environments. In International Symposium on Foundations of Software Engineering, pages 235–245, 2014.
[24]
S. Erdweg, M. Lichter, and W. Manuel. A sound and optimal incremental build system with dynamic dependencies. In Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 89–106, 2015.
[25]
H. Esfahani, J. Fietz, Q. Ke, A. Kolomiets, E. Lan, E. Mavrinac, W. Schulte, N. Sanches, and S. Kandula. CloudBuild: Microsoft’s distributed and caching build service. pages 11–20, 2016.
[26]
T. Gamblin, M. LeGendre, M. R. Collette, G. L. Lee, A. Moody, B. R. de Supinski, and S. Futral. The Spack package manager: Bringing order to HPC software chaos. In International Conference for High Performance Computing, Networking, Storage and Analysis, pages 40:1–40:12, 2015.
[27]
Z. Gao, Y. Liang, M. B. Cohen, A. M. Memon, and Z. Wang. Making system user interactive tests repeatable: When and what should we control? In International Conference on Software Engineering, pages 55–65, 2015.
[28]
GitLab continuous integration. https://about.gitlab. com/gitlab-ci/.
[29]
M. Gligoric, L. Eloussi, and D. Marinov. Practical regression test selection with dynamic file dependencies. In International Symposium on Software Testing and Analysis, pages 211–222, 2015.
[30]
M. Gligoric, W. Schulte, C. Prasad, D. van Velzen, I. Narasamdya, and B. Livshits. Automated migration of build scripts using dynamic analysis and search-based refactoring. In Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 599–616, 2014.
[31]
B. Godlin and O. Strichman. Regression verification: proving the equivalence of similar programs. Journal of Software Testing, Verification and Reliability, 23(3):241–258, 2013.
[32]
Gradle build tool. https://gradle.org.
[33]
T. A. Henzinger, R. Jhala, R. Majumdar, and M. A. A. Sanvido. Extreme model checking. In Verification: Theory and Practice, pages 332–358, 2003.
[34]
M. Hicks, J. T. Moore, and S. Nettles. Dynamic software updating. In Conference on Programming Language Design and Implementation, pages 13–23, 2001.
[35]
M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig. Usage, costs, and benefits of continuous integration in open-source projects. In Automated Software Engineering, 2016. To appear.
[36]
It’s Travis CI’s 5th birthday, let’s celebrate with numbers! https://blog.travis-ci.com/ 2016-02-05-happy-fifth-birthday-travis-ci.
[37]
Java language and virtual machine specifications. https://docs.oracle.com/javase/specs.
[38]
Java Web Start. http://www.oracle.com/technetwork/ java/javase/javawebstart/index.html.
[39]
S. Kamin, L. Clausen, and A. Jarvis. Jumbo: Run-time code generation for Java and its applications. In International Symposium on Code Generation and Optimization: Feedback-directed and Runtime Optimization, pages 48–56, 2003.
[40]
R. Kapitza and F. J. Hauck. DLS: a CORBA service for dynamic loading of code. In On The Move to Meaningful Internet Systems: CoopIS, DOA, and ODBASE, pages 1333–1350. 2003.
[41]
D. C. Kung, J. Gao, P. Hsia, J. Lin, and Y. Toyoshima. Class firewall, test order, and regression testing of object-oriented programs. Journal of Object-Oriented Programming, 8(2):51–65, 1995.
[42]
H. Kurshan, R. H. Hardin, R. P. Kurshan, K. L. Mcmillan, J. A. Reeds, and N. J. A. Sloane. Efficient regression verification. In International Workshop on Discrete Event Systems, pages 147–150, 1996.
[43]
S. Liang and G. Bracha. Dynamic class loading in the Java virtual machine. SIGPLAN Notices, 33(10):36–44, 1998.
[44]
Q. Luo, F. Hariri, L. Eloussi, and D. Marinov. An empirical analysis of flaky tests. In International Symposium on Foundations of Software Engineering, pages 643–653, 2014.
[45]
S. McIntosh, B. Adams, and A. E. Hassan. The evolution of Java build systems. Empirical Software Engineering, 17:578–608, 2012.
[46]
B. Mitchell and S. Mancoridis. On the automatic modularization of software systems using the Bunch tool. Transactions on Software Engineering, 32(3):193–208, 2006.
[47]
A. Neitsch, K. Wong, and M. Godfrey. Build system issues in multilanguage software. In International Conference on Software Maintenance, pages 140–149, 2012.
[48]
Oracle. pack200 specification. http://docs.oracle.com/ javase/7/docs/technotes/guides/pack200/pack-spec. html.
[49]
ProGuard. http://proguard.sourceforge.net.
[50]
M. Pukall, A. Grebhahn, R. Schröter, C. Kästner, W. Cazzola, and S. Götz. JavAdaptor: Unrestricted dynamic software updates for Java. In International Conference on Software Engineering, pages 989–991, 2011.
[51]
T. Ritzau and J. Andersson. Dynamic deployment of Java applications. In Java for Embedded Systems Workshop, 2000.
[52]
C. Sadowski, J. van Gogh, C. Jaspan, E. Söderberg, and C. Winter. Tricorder: Building a program analysis ecosystem. In International Conference on Software Engineering, pages 598–608, 2015.
[53]
sbt - the interactive build tool. http://www.scala-sbt. org.
[54]
SCons: A software construction tool. http://www. scons.org.
[55]
A. Shi, T. Yung, A. Gyori, and D. Marinov. Comparing and combining test-suite reduction and regression test selection. In International Symposium on Foundations of Software Engineering, pages 237–247, 2015.
[56]
Speeding up the build. http://docs.travis-ci.com/user/ speeding-up-the-build.
[57]
S. Subramanian, M. Hicks, and K. S. McKinley. Dynamic software updates: A VM-centric approach. In Conference on Programming Language Design and Implementation, pages 1–12, 2009.
[58]
The Apt package manager. https://wiki.debian.org/ Apt.
[59]
The class file format. https://docs.oracle.com/javase/ specs/jvms/se7/html/jvms-4.html.
[60]
Travis CI - test and deploy with confidence. https:// travis-ci.com/.
[61]
Travis CI - issue 1441. https://github.com/travis-ci/ travis-ci/issues/1441.
[62]
Travis CI - issue 189. https://github.com/yegor256/ thindeck/issues/189.
[63]
M. Vakilian, R. Sauciuc, J. D. Morgenthaler, and V. Mirrokni. Automated decomposition of build targets. In International Conference on Software Engineering, pages 123–133, 2015.
[64]
S. Vinoski. CORBA: integrating diverse applications within distributed heterogeneous environments. Communications Magazine, 35(2):46–55, 1997.
[65]
X. Xia, X. Zhou, D. Lo, and X. Zhao. An empirical study of bugs in software build systems. In International Conference on Quality Software, pages 200–203, 2013.
[66]
S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. Journal of Software Testing, Verification and Reliability, 22(2):67–120, 2012.
[67]
Y. Yu, H. Dayani-Fard, and J. Mylopoulos. Removing false code dependencies to speedup software build processes. In Conference of the Centre for Advanced Studies on Collaborative Research, pages 343–352, 2003.

Cited By

View all
  • (2024)Commit Artifact Preserving Build PredictionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680356(1236-1248)Online publication date: 11-Sep-2024
  • (2024)PIPELINEASCODE: A CI/CD Workflow Management System through Configuration Files at ByteDance2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00109(1011-1022)Online publication date: 12-Mar-2024
  • (2023)HybridCISave: A Combined Build and Test Selection Approach in Continuous IntegrationACM Transactions on Software Engineering and Methodology10.1145/357603832:4(1-39)Online publication date: 26-May-2023
  • Show More Cited By

Index Terms

  1. Build system with lazy retrieval for Java projects

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering
    November 2016
    1156 pages
    ISBN:9781450342186
    DOI:10.1145/2950290
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 01 November 2016

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Build system
    2. continuous integration service

    Qualifiers

    • Research-article

    Conference

    FSE'16
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 17 of 128 submissions, 13%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)14
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Commit Artifact Preserving Build PredictionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680356(1236-1248)Online publication date: 11-Sep-2024
    • (2024)PIPELINEASCODE: A CI/CD Workflow Management System through Configuration Files at ByteDance2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00109(1011-1022)Online publication date: 12-Mar-2024
    • (2023)HybridCISave: A Combined Build and Test Selection Approach in Continuous IntegrationACM Transactions on Software Engineering and Methodology10.1145/357603832:4(1-39)Online publication date: 26-May-2023
    • (2023)Coverage-Based Debloating for Java BytecodeACM Transactions on Software Engineering and Methodology10.1145/354694832:2(1-34)Online publication date: 4-Apr-2023
    • (2023)Understanding the quality and evolution of Android app build systemsJournal of Software: Evolution and Process10.1002/smr.2602Online publication date: 6-Aug-2023
    • (2022)BuildSonic: Detecting and Repairing Performance-Related Configuration Smells for Continuous Integration BuildsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556923(1-13)Online publication date: 10-Oct-2022
    • (2021)Reducing cost in continuous integration with a collection of build selection approachesProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3473103(1650-1654)Online publication date: 20-Aug-2021
    • (2021)What helped, and what did not?Proceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00031(213-225)Online publication date: 22-May-2021
    • (2020)A model for detecting faults in build specificationsProceedings of the ACM on Programming Languages10.1145/34282124:OOPSLA(1-30)Online publication date: 13-Nov-2020
    • (2020)A cost-efficient approach to building in continuous integrationProceedings of the ACM/IEEE 42nd International Conference on Software Engineering10.1145/3377811.3380437(13-25)Online publication date: 27-Jun-2020
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media