Joseph Hejderup
ABSTRACT
Development teams are increasingly investing in automating the updating of third-party libraries to limit the patch time of zero-day exploits such as the Equifax breach. GitHub bots such as Dependabot and Renovate build such functionality by leveraging existing test infrastructure in repositories to test and evaluate new library updates. However, two recent studies suggest that test suites in projects lack effectiveness and coverage to reliably find regressions in third-party libraries. Adequate test coverage and effectiveness are critical in discovering new vulnerabilities and weaknesses from third-party libraries. The recent Log4Shell incident exemplifies this, as projects will likely not have adequate tests for logging libraries. This position paper discusses the weaknesses and challenges of current testing practices and techniques from a supply chain security perspective. We highlight two key challenges that researchers and practitioners need to address: (1) the lack of resources and best practices for testing the uses of third-party libraries and (2) enhancing the reliability of automating library updates.
- Taylor Armerding. 2022. Open Source News from the 2022 OSSRA Report. https://thenewstack.io/open-source-news-from-the-2022-ossra-report/. (2022). (Accessed on 05/08/2022).Google Scholar
- Christopher Bogart, Christian Kästner, James Herbsleb, and Ferdian Thung. 2016. How to break an API: cost negotiation and community values in three software ecosystems. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 109--120.Google ScholarDigital Library
- Thomas Claburn. 2018. Check your repos... Crypto-coin-stealing code sneaks into fairly popular NPM lib (2m downloads per week). https://www.theregister. com/2018/11/26/npm_repo_bitcoin_stealer/. (2018). (Accessed on 05/08/2022).Google Scholar
- Yaniv David, Xudong Sun, Raphael J Sofaer, Aditya Senthilnathan, Junfeng Yang, Zhiqiang Zuo, Guoqing Harry Xu, Jason Nieh, and Ronghui Gu. 2022. {UPGRADVISOR}: Early Adopting Dependency Updates Using Hybrid Program Analysis and Hardware Tracing. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 751--767.Google Scholar
- Alexandre Decan, Tom Mens, and Philippe Grosjean. 2019. An empirical comparison of dependency network evolution in seven software packaging ecosystems. Empirical Software Engineering 24, 1 (2019), 381--416.Google ScholarDigital Library
- Dependabot. 2022. Automated dependency updates. https://dependabot.com/. (2022). (Accessed on 17/04/2022).Google Scholar
- Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416--419.Google ScholarDigital Library
- Joseph Hejderup and Georgios Gousios. 2022. Can we trust tests to automate dependency updates? a case study of java projects. Journal of Systems and Software 183 (2022), 111097.Google ScholarCross Ref
- William C Hetzel and Bill Hetzel. 1988. The complete guide to software testing. QED Information Sciences Wellesley, MA.Google Scholar
- Nathan P Kropp, Philip J Koopman, and Daniel P Siewiorek. 1998. Automated robustness testing of off-the-shelf software components. In Digest of Papers. Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing (Cat. No. 98CB36224). IEEE, 230--239.Google ScholarDigital Library
- Leonardo Mariani, Sofia Papagiannakis, and Mauro Pezze. 2007. Compatibility and regression testing of COTS-component-based software. In 29th International Conference on Software Engineering (ICSE'07). IEEE, 85--95.Google ScholarDigital Library
- Samim Mirhosseini and Chris Parnin. 2017. Can automated pull requests encourage software developers to upgrade out-of-date dependencies?. In 2017 32nd IEEE/ACM international conference on automated software engineering (ASE). IEEE, 84--94.Google ScholarCross Ref
- Ons Mlouki, Foutse Khomh, and Giuliano Antoniol. 2016. On the detection of licenses violations in the android ecosystem. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. IEEE, 382--392.Google ScholarCross Ref
- Glenford J Myers, Corey Sandler, and Tom Badgett. 2011. The art of software testing. John Wiley & Sons.Google ScholarDigital Library
- Mitchell Olsthoorn, Pouria Derakhshanfar, and Annibale Panichella. 2021. Hybrid Multi-level Crossover for Unit Test Case Generation. In International Symposium on Search Based Software Engineering. Springer, 72--86.Google Scholar
- Renovate. 2022. Automated dependency management. https://renovatebot.com/. (2022). (Accessed on 26/07/2022).Google Scholar
- Kostya Serebryany. 2017. {OSS-Fuzz}-Google's continuous fuzzing service for open source software. (2017).Google Scholar
- Marat Valiev, Bogdan Vasilescu, and James Herbsleb. 2018. Ecosystem-level determinants of sustained activity in open-source projects: A case study of the PyPI ecosystem. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 644--655.Google ScholarDigital Library
- James A Whittaker. 2002. How to Break Software: A Practical Guide to Testing with Cdrom. Addison-Wesley Longman Publishing Co., IncGoogle Scholar
Index Terms
- On the Use of Tests for Software Supply Chain Threats
Recommendations
Towards Using Source Code Repositories to Identify Software Supply Chain Attacks
CCS '20: Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications SecurityIncreasing popularity of third-party package repositories, like NPM, PyPI, or RubyGems, makes them an attractive target for software supply chain attacks. By injecting malicious code into legitimate packages, attackers were known to gain more than 100,...
Organization of Testing Activities in Norwegian Software Companies
ICSTW '13: Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation WorkshopsSoftware testing plays a major role in determining the quality of software products. Testing activities are influenced by the available methods and tools, but also by the non-technical aspects of the software development process. The scope of the study ...
Toward the characterization of software testing practices in South America: looking at Brazil and Uruguay
Software testing is an important activity in the software development life cycle. Several previous studies reported the results of surveys on software testing practices among practitioners from different countries. In this paper, we analyze these ...
Comments