ABSTRACT
Modern software projects rely on build systems and build scripts to assemble executable artifacts correctly and efficiently. However, developing build scripts is error-prone. Dependency-related errors in build scripts, mainly including missing dependencies and redundant dependencies, are common in various kinds of software projects. These errors lead to build failures, incorrect build results or poor performance in incremental or parallel builds. To detect such errors, various techniques are proposed and suffer from low efficiency and high false positive problems, due to the deficiency of the underlying dependency graphs. In this work, we design a new dependency graph, the unified dependency graph (UDG), which leverages both static and dynamic information to uniformly encode the declared and actual dependencies between build targets and files. The construction of UDG facilitates the efficient and precise detection of dependency errors via simple graph traversals. We implement the proposed approach as a tool, VeriBuild, and evaluate it on forty-two well-maintained open-source projects. The experimental results show that, without losing precision, VeriBuild incurs 58.2% less overhead than the state-of-the-art approach. By the time of writing, 398 detected dependency issues have been confirmed by the developers.
- Bram Adams, Kris De Schutter, Herman Tromp, and Wolfgang De Meuter. 2008. The evolution of the Linux build system. Electronic Communications of the EASST ( 2008 ). https://doi.org/10.14279/tuj. eceasst.8.115.119 Google ScholarCross Ref
- Bram Adams, Herman Tromp, Kris De Schutter, and Wolfgang De Meuter. 2007. Design recovery and maintenance of build systems. In IEEE International Conference on Software Maintenance (ICSM). https://doi.org/10.1109/ICSM. 2007. 4362624 Google ScholarCross Ref
- Shivali Agarwal, Rajkishore Barik, Vivek Sarkar, and Rudrapatna K. Shyamasundar. 2007. May-happen-in-parallel analysis of X10 programs. In Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPOPP). https://doi.org/10.1145/1229428.1229471 Google ScholarDigital Library
- Jafar Al-Kofahi, Hung Viet Nguyen, and Tien N. Nguyen. 2014. Fault localization for make-based build crashes. In Proceedings-30th International Conference on Software Maintenance and Evolution (ICSME). https://doi.org/10.1109/ICSME. 2014.87 Google ScholarDigital Library
- Jafar Al-Kofahi, Hung Viet Nguyen, and Tien N. Nguyen. 2014. Fault localization for build code errors in makefiles. In 36th International Conference on Software Engineering, ICSE Companion 2014-Proceedings. https://doi.org/10.1145/2591062. 2591135 Google ScholarDigital Library
- Cor Paul Bezemer, Shane McIntosh, Bram Adams, Daniel M. German, and Ahmed E. Hassan. 2017. An empirical study of unspecified dependencies in make-based build systems. Empirical Software Engineering ( 2017 ). https: //doi.org/10.1007/s10664-017-9510-8 Google ScholarDigital Library
- Electric Cloud. 2020. ElectricAccelerator EMake: Speeds Up Builds and Tests. https://electric-cloud.com/plugins/directory/p/emake/ [Online; accessed 25-Aug2019 ].Google Scholar
- Automake Contributors. 2012. GNU Autotools. https://www.gnu.org/software/ automake [Online; accessed 25-Aug-2019].Google Scholar
- Drake. 2020. Build Cop. https://drake.mit.edu/buildcop.html [Online; accessed 25-May-2020].Google Scholar
- Stuart I. Feldman. 1979. Make-a program for maintaining computer programs. Software: Practice and Experience ( 1979 ). https://doi.org/10.1002/spe.4380090402 Google ScholarCross Ref
- Milos Gligoric, Wolfram Schulte, Chandra Prasad, Danny van Velzen, Iman Narasamdya, and Benjamin Livshits. 2014. Automated migration of build scripts using dynamic analysis and search-based refactoring. ACM SIGPLAN Notices ( 2014 ). https://doi.org/10.1145/2714064.2660239 Google ScholarDigital Library
- Carl A. Gunter. 1996. Abstracting dependencies between software configuration items. In Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE). https://doi.org/10.1145/250707.239129 Google ScholarDigital Library
- Ryan Hardt and Ethan V. Munson. 2013. Ant build maintenance with formiga. In 2013 1st International Workshop on Release Engineering, RELENG 2013-Proceedings. https://doi.org/10.1109/RELENG. 2013.6607690 Google ScholarCross Ref
- Foyzul Hassan and Xiaoyin Wang. 2018. HireBuild: An automatic approach to history-driven repair of build scripts. In Proceedings of the 40th International Conference on Software Engineering (ICSE). IEEE/ACM. https://doi.org/10.1145/ 3180155.3180181 Google ScholarDigital Library
- Lorin Hochstein and Yang Jiao. 2011. The cost of the build tax in scientific software. In International Symposium on Empirical Software Engineering and Measurement (ESEM).Google ScholarDigital Library
- International Business Machines Corporation (IBM). 2020. IBM Rational Clearcase. https://www.ibm.com/us-en/marketplace/rational-clearcase [Online; accessed 25-Aug-2019].Google Scholar
- Google Inc. 2019. Actual and declared dependencies. https://docs.bazel.build/ versions/master/build-ref.html#actual_and_declared_dependencies [Online; accessed 25-Aug-2019].Google Scholar
- Google Inc. 2019. Bazel-a fast, scalable, multi-language and extensible build system. https://bazel.build/ [Online; accessed 25-May-2020].Google Scholar
- Google Inc. 2019. Bazel Query. https://docs.bazel.build/versions/master/queryhow-to. htmle [Online; accessed 25-Aug-2019].Google Scholar
- Lim James. 2019. Combating Flaky Builds. https://medium.com/@jimjh/ combating-flaky-builds-f8aaa9ccd29a [Online; accessed 25-Aug-2019].Google Scholar
- Petr Machata Juan Cespedes. 2019. ltrace. http://man7.org/linux/manpages/man1/ltrace.1.html [Online; accessed 25-May-2020].Google Scholar
- Nándor Licker and Andrew Rice. 2019. Detecting incorrect build rules. In Proceedings of the 41st International Conference on Software Engineering (ICSE). IEEE, 1234-1244. https://doi.org/10.1109/ICSE. 2019.00125 Google ScholarDigital Library
- Yiling Lou, Junjie Chen, Lingming Zhang, Dan Hao, and Lu Zhang. 2019. Historydriven build failure fixing: How far are we?. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). https://doi.org/10.1145/3293882.3330578 Google ScholarDigital Library
- Christian Macho, Shane McIntosh, and Martin Pinzger. 2018. Automatically repairing dependency-related build breakage. In 25th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). https: //doi.org/10.1109/SANER. 2018.8330201 Google ScholarCross Ref
- GNU Make Manual. 2019. Auto-Dependency Generation. ftp://ftp.gnu.org/oldgnu/Manuals/make-3.77/html_node/make_43.html [Online; accessed 25-Aug2019 ].Google Scholar
- GNU Make Manual. 2019. Prerequisite Types. https://www.gnu.org/software/ make/manual/html_node/Prerequisite-Types. html [Online; accessed 25-Aug2019].Google Scholar
- Ken Martin and Bill Hofman. 2010. Mastering CMake: a cross-platform build system. Kitware.Google Scholar
- Friedemann Mattern. 1989. Virtual Time and Global States of Distributed Systems. Event London ( 1989 ). https://doi.org/10.1.1.47.7435Google Scholar
- Bill McCloskey. 2019. memoize. https://github.com/kgaughan/memoize.py [Online; accessed 25-May-2020].Google Scholar
- Shane McIntosh, Bram Adams, Meiyappan Nagappan, and Ahmed E. Hassan. 2014. Mining co-change information to understand when build changes are necessary. In Proceedings-30th International Conference on Software Maintenance and Evolution (ICSME). https://doi.org/10.1109/ICSME. 2014.46 Google ScholarDigital Library
- Microsoft. 2020. NMAKE Reference. https://docs.microsoft.com/en-us/cpp/ build/reference/nmake-reference?view= vs-2019 [Online; accessed 25-May-2020].Google Scholar
- Eric S Raymond. 2003. The art of Unix programming. Addison-Wesley Professional.Google ScholarDigital Library
- Zhilei Ren, He Jiang, Jifeng Xuan, and Zijiang Yang. 2018. Automated Localization for Unreproducible Builds. In Proceedings of the 40th International Conference on Software Engineering ( ICSE) (Gothenburg, Sweden). Association for Computing Machinery, New York, NY, USA, 71-81. https://doi.org/10.1145/3180155.3180224 Google ScholarDigital Library
- Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum, Edward Aftandilian, and Robert Bowdidge. 2014. Programmers' build errors: A case study (at google). In Proceedings-International Conference on Software Engineering (ICSE). https: //doi.org/10.1145/2568225.2568255 Google ScholarDigital Library
- Mike Shal. 2009. Build system rules and algorithms. Published online ( 2009 ). Retrieved July 18 ( 2009 ), 2013. http://gittup.org/tup/build_system_rules_and_algorithms.pdfGoogle Scholar
- Ahmed Tamrawi, Hoan Anh Nguyen, Hung Viet Nguyen, and Tien N. Nguyen. 2012. Build Code Analysis with Symbolic Evaluation. In Proceedings of the 34th International Conference on Software Engineering ( ICSE) (Zurich, Switzerland). IEEE Press, 650-660.Google Scholar
- Ahmed Tamrawi, Hoan Anh Nguyen, Hung Viet Nguyen, and Tien N Nguyen. 2012. SYMake: a build code analysis and refactoring tool for makefiles. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (ASE). ACM, 366-369.Google ScholarDigital Library
- Brush Technology. 2019. fabricate. https://github.com/brushtechnology/fabricate [Online; accessed 25-May-2020].Google Scholar
- Mohsen Vakilian, Raluca Sauciuc, J. David Morgenthaler, and Vahab Mirrokni. 2015. Automated Decomposition of Build Targets. In Proceedings of the 37th International Conference on Software Engineering-Volume 1 ( ICSE ) (Florence, Italy). IEEE Press, 123-133.Google ScholarDigital Library
- Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. 2018. Un-break My Build: Assisting Developers with Build Repair Hints. 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC) ( 2018 ), 41-4110.Google Scholar
- VestaSys. 2020. Vesta Configuration Management System. http://www.vestasys. org/ [Online; accessed 25-May-2020].Google Scholar
- Wikipedia contributors. 2020. List of NP-complete problems. https://en.wikipedia. org/w/index.php?title=List_of_NP-complete_problems &oldid=957698266 [Online; accessed 25-May-2020].Google Scholar
- Xin Xia, David Lo, Xinyu Wang, and Bo Zhou. 2014. Build system analysis with link prediction. In Proceedings of the ACM Symposium on Applied Computing (SAC). https://doi.org/10.1145/2554850.2555134 Google ScholarDigital Library
- Bo Zhou, Xin Xia, David Lo, and Xinyu Wang. 2014. Build predictor: More accurate missed dependency prediction in build configuration files. In Proceedings-International Computer Software and Applications Conference (COMPSAC). https://doi.org/10.1109/COMPSAC. 2014.12 Google ScholarDigital Library
Index Terms
- Escaping dependency hell: finding build dependency errors with the unified dependency graph
Recommendations
Fixing dependency errors for Python build reproducibility
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and AnalysisSoftware reproducibility is important for re-usability and the cumulative progress of research. An important manifestation of unreproducible software is the changed outcome of software builds over time. While enhancing code reuse, the use of open-source ...
Accelerating Build Dependency Error Detection via Virtual Build
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software EngineeringBuild scripts play an important role in transforming the source code into executable artifacts. However, the development of build scripts is typically error-prone. As one kind of the most prevalent errors in build scripts, the dependency-related errors, ...
Relational Dependency Networks
Recent work on graphical models for relational data has demonstrated significant improvements in classification and inference when models represent the dependencies among instances. Despite its use in conventional statistical models, the assumption of ...
Comments