skip to main content
10.1145/3106237.3106297acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Regression test selection across JVM boundaries

Published: 21 August 2017 Publication History

Abstract

Modern software development processes recommend that changes be integrated into the main development line of a project multiple times a day. Before a new revision may be integrated, developers practice regression testing to ensure that the latest changes do not break any previously established functionality. The cost of regression testing is high, due to an increase in the number of revisions that are introduced per day, as well as the number of tests developers write per revision. Regression test selection (RTS) optimizes regression testing by skipping tests that are not affected by recent project changes. Existing dynamic RTS techniques support only projects written in a single programming language, which is unfortunate knowing that an open-source project is on average written in several programming languages.
We present the first dynamic RTS technique that does not stop at predefined language boundaries. Our technique dynamically detects, at the operating system level, all file artifacts a test depends on. Our technique is, hence, oblivious to the specific means the test uses to actually access the files: be it through spawning a new process, invoking a system call, invoking a library written in a different language, invoking a library that spawns a process which makes a system call, etc. We also provide a set of extension points which allow for a smooth integration with testing frameworks and build systems. We implemented our technique in a tool called RTSLinux as a loadable Linux kernel module and evaluated it on 21 Java projects that escape JVM by spawning new processes or invoking native code, totaling 2,050,791 lines of code. Our results show that RTSLinux, on average, skips 74.17% of tests and saves 52.83% of test execution time compared to executing all tests.

References

[1]
Ant Home Page. http://ant.apache.org.
[2]
Apache Hadoop Home Page. http://hadoop.apache.org.
[3]
Build in the Cloud. http://google-engtools.blogspot.com/2011/08/ build-in-cloud-how-build-system-works.html.
[4]
JUnit Task. https://ant.apache.org/manual/Tasks/junit.html.
[5]
Linus Torvalds - Moving to Linux 4.0. https://plus.google.com/+LinusTorvalds/ posts/jmtzzLiiejc.
[6]
Linux Loadable Kernel Module HOWTO. http://tldp.org/HOWTO/ Module-HOWTO.
[7]
pendulum - Python datetimes made easy. https://github.com/sdispater/ pendulum.
[8]
perf trace & vfs_getname. http://www.spinics.net/lists/linux-perf-users/ msg02975.html.
[9]
SCons. http://www.scons.org.
[10]
strace - trace system calls and signals. http://linux.die.net/man/1/strace.
[11]
Travis CI - Test and Deploy Your Code with Confidence. https://travis-ci.org.
[12]
Tup. http://gittup.org/tup.
[13]
Elaine Angelino, Daniel Yamins, and Margo I. Seltzer. 2010. StarFlow: A Script-Centric Data Analysis Environment. In International Provenance and Annotation Workshop. 236–250.
[14]
Thomas Ball. 1998. On the Limit of Control Flow Analysis for Regression Test Selection. In International Symposium on Software Testing and Analysis. 134–142.
[15]
Adam Bates, Dave Tian, Kevin R. B. Butler, and Thomas Moyer. 2015. Trustworthy Whole-system Provenance for the Linux Kernel. In USENIX Conference on Security Symposium. 319–334.
[16]
Bazel. http://bazel.io/.
[17]
Boris Beizer. 1990. Software Testing Techniques (2nd Ed.). Van Nostrand Reinhold Co., New York, NY, USA.
[18]
Jonathan Bell and Gail E. Kaiser. 2014. Unit test virtualization with VMVM. In International Conference on Software Engineering. 550–561.
[19]
Swarnendu Biswas, Rajib Mall, Manoranjan Satpathy, and Srihari Sukumaran. 2011. Regression Test Selection Techniques: A Survey. Informatica (Slovenia) 35, 3 (2011), 289–321.
[20]
Ahmet Celik, Alex Knaust, Aleksandar Milicevic, and Milos Gligoric. 2016. Build System with Lazy Retrieval for Java Projects. In International Symposium on Foundations of Software Engineering. 643–654.
[21]
Yih-Farn Chen, David S. Rosenblum, and Kiem-Phong Vo. 1994. TestTube: A System for Selective Regression Testing. In International Conference on Software Engineering. 211–220.
[22]
Pavan Kumar Chittimalli and Mary Jean Harrold. 2007. Re-computing Coverage Information to Assist Regression Testing. In International Conference on Software Maintenance. 164–173.
[23]
Maria Christakis, K. Rustan M. Leino, and Wolfram Schulte. 2014. Formalizing and Verifying a Modern Build Language. In International Symposium on Formal Methods. 643–657.
[24]
Cleaning directories after each test run, to prevent repository pollution. https: //github.com/apache/commons-io/pull/13.
[25]
Cleaning directories after each test run, to prevent repository pollution. https: //issues.apache.org/jira/browse/CONFIGURATION-638.
[26]
Sebastian Elbaum, Gregg Rothermel, and John Penix. 2014. Techniques for Improving Regression Testing in Continuous Integration Development Environments. In International Symposium on Foundations of Software Engineering. 235–245.
[27]
Emelie Engström and Per Runeson. 2010. A Qualitative Survey of Regression Testing Practices. In Product-Focused Software Process Improvement. 3–16.
[28]
Emelie Engström, Per Runeson, and Mats Skoglund. 2010. A Systematic Review on Regression Test Selection Techniques. Journal of Information and Software Technology 52, 1 (2010), 14–30.
[29]
Emelie Engström, Mats Skoglund, and Per Runeson. 2008. Empirical evaluations of regression test selection techniques: a systematic review. In International Symposium on Empirical Software Engineering and Measurement. 22–31.
[30]
Eradicating Non-Determinism in Tests. http://martinfowler.com/articles/ nonDeterminism.html.
[31]
Hamed Esfahani, Jonas Fietz, Qi Ke, Alexei Kolomiets, Erica Lan, Erik Mavrinac, Wolfram Schulte, Newton Sanches, and Srikanth Kandula. 2016. CloudBuild: Microsoft’s Distributed and Caching Build Service. In International Conference on Software Engineering, Software Engineering in Practice. 11–20.
[32]
Fabricate. https://github.com/SimonAlfie/fabricate.
[33]
Erich Gamma, John Vlissides, Ralph Johnson, and Richard Helm. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.
[34]
Ashish Gehani and Dawood Tariq. 2012. SPADE: Support for Provenance Auditing in Distributed Environments. In International Middleware Conference. 101–120.
[35]
Milos Gligoric. 2015. Regression Test Selection: Theory and Practice. Ph.D. Dissertation. The University of Illinois at Urbana-Champaign.
[36]
Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Ekstazi: Lightweight Test Selection. In International Conference on Software Engineering, Demo. 713– 716.
[37]
Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Practical Regression Test Selection with Dynamic File Dependencies. In International Symposium on Software Testing and Analysis. 211–222.
[38]
Milos Gligoric, Wolfram Schulte, Chandra Prasad, Danny van Velzen, Iman Narasamdya, and Benjamin Livshits. 2014. Automated Migration of Build Scripts using Dynamic Analysis and Search-Based Refactoring. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 599–616.
[39]
Gradle Build Tool - Modern Open Source Build Automation. http://gradle.org.
[40]
Philip J. Guo and Dawson Engler. 2011. Using Automatic Persistent Memoization to Facilitate Data Analysis Scripting. In International Symposium on Software Testing and Analysis. 287–297.
[41]
Alex Gyori, August Shi, Farah Hariri, and Darko Marinov. 2015. Reliable testing: detecting state-polluting tests to prevent test dependency. In International Symposium on Software Testing and Analysis. 223–233.
[42]
Ramzi A. Haraty, Nash’at Mansour, and Bassel Daou. 2001. Regression testing of database applications. In Symposium on Applied Computing. 285–289.
[43]
Ramzi A. Haraty, Nashat Mansour, and Bassel A. Daou. 2004. Regression test selection for database applications. Advanced Topics in Database Research 3 (2004), 141–165.
[44]
Jean Hartmann. 2012. 30 Years of Regression Testing: Past, Present and Future. In Pacific Northwest Software Quality Conference. 119–126.
[45]
Allan Heydon, Roy Levin, Timothy Mann, and Yuan Yu. 2002. The Vesta Software Configuration Management System. Research Report. http://www.hpl.hp.com/ techreports/Compaq-DEC/SRC-RR-177.pdf.
[46]
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects. In Automated Software Engineering. 426–437.
[47]
JNI APIs and Developer Guides. https://docs.oracle.com/javase/8/docs/technotes/ guides/jni.
[48]
KernelGitGuide. https://wiki.ubuntu.com/Kernel/Dev/KernelGitGuide.
[49]
David Chenho Kung, Jerry Gao, Pei Hsia, Jeremy Lin, and Yasufumi Toyoshima. 1995. Class Firewall, Test Order, and Regression Testing of Object-Oriented Programs. Journal of Object-Oriented Programming 8, 2 (1995), 51–65.
[50]
Kyu Hyung Lee, Xiangyu Zhang, and Dongyan Xu. 2013. High Accuracy Attack Provenance via Binary-based Execution Partition. In Network and Distributed System Security Symposium.
[51]
Owolabi Legunsen, Farah Hariri, August Shi, Yafeng Lu, Lingming Zhang, and Darko Marinov. 2016. An Extensive Study of Static Regression Test Selection in Modern Software Evolution. In International Symposium on Foundations of Software Engineering. 883–894.
[52]
Hareton K. N. Leung and Lee White. 1989. Insights into regression testing. In International Conference on Software Maintenance. 60–69.
[53]
Sheng Liang. 1999. Java Native Interface : Programmer’s Guide and Specification. Addison-Wesley Longman.
[54]
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An Empirical Analysis of Flaky Tests. In International Symposium on Foundations of Software Engineering. 643–653.
[55]
Peter Macko and Margo Seltzer. 2012. A General-purpose Provenance Library. In USENIX Conference on Theory and Practice of Provenance. 6–6.
[56]
Apache Maven. https://maven.apache.org.
[57]
Philip Mayer and Alexander Bauer. 2015. An Empirical Analysis of the Utilization of Multiple Programming Languages in Open Source Projects. In International Conference on Evaluation and Assessment in Software Engineering. 1–10.
[58]
Shane Mcintosh, Bram Adams, and Ahmed E. Hassan. 2012. The Evolution of Java Build Systems. Empirical Software Engineering 17, 4–5 (2012), 578–608.
[59]
Memoize. https://github.com/kgaughan/memoize.py.
[60]
Atif M. Memon and Myra B. Cohen. 2013. Automated Testing of GUI Applications: Models, Tools, and Controlling Flakiness. In International Conference on Software Engineering. 1479–1480.
[61]
Brian S. Mitchell and Spiros Mancoridis. 2006. On the automatic modularization of software systems using the Bunch tool. Transactions on Software Engineering 32, 3 (2006), 193–208.
[62]
Kivanç Muşlu, Bilge Soran, and Jochen Wuttke. 2011. Finding Bugs by Isolating Unit Tests. In International Symposium on Foundations of Software Engineering. 496–499.
[63]
Kiran-Kumar Muniswamy-Reddy, Uri Braun, David A. Holland, Peter Macko, Diana Maclean, Daniel Margo, Margo Seltzer, and Robin Smogor. 2009. Layering in Provenance Systems. In Conference on USENIX Annual Technical Conference. 10–10.
[64]
Agastya Nanda, Senthil Mani, Saurabh Sinha, Mary Jean Harrold, and Alessandro Orso. 2011. Regression Testing in the Presence of Non-code Changes. In International Conference on Software Testing, Verification, and Validation. 21–30.
[65]
Alexander Neundorf. Why the KDE project switched to CMake - and how. http://lwn.net/Articles/188693.
[66]
ESEC/FSE’17, September 4–8, 2017, Paderborn, Germany Ahmet Celik, Marko Vasic, Aleksandar Milicevic, and Milos Gligoric
[67]
Alessandro Orso and Gregg Rothermel. 2014. Software Testing: A Research Travelogue (2000–2014). In Future of Software Engineering. 117–132.
[68]
Alessandro Orso, Nanjuan Shi, and Mary Jean Harrold. 2004. Scaling Regression Testing to Large Software Systems. In International Symposium on Foundations of Software Engineering. 241–251.
[69]
Baishakhi Ray, Daryl Posnett, Vladimir Filkov, and Premkumar Devanbu. 2014. A Large Scale Study of Programming Languages and Code Quality in GitHub. In International Symposium on Foundations of Software Engineering. 155–165.
[70]
Xiaoxia Ren, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia Chesley. 2004. Chianti: A Tool for Change Impact Analysis of Java Programs. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 432–448.
[71]
Gregg Rothermel and Mary Jean Harrold. 1996. Analyzing Regression Test Selection Techniques. Transactions on Software Engineering 22, 8 (1996), 529– 551.
[72]
Gregg Rothermel and Mary Jean Harrold. 1997. A safe, efficient regression test selection technique. Transactions on Software Engineering and Methodology 6, 2 (1997), 173–210.
[73]
August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, and Darko Marinov. 2014. Balancing Trade-offs in Test-suite Reduction. In International Symposium on Foundations of Software Engineering. 246–256.
[74]
Mats Skoglund and Per Runeson. 2005. A case study of the class firewall regression test selection technique on a large scale distributed software system. In International Symposium on Empirical Software Engineering. 74–83.
[75]
Mats Skoglund and Per Runeson. 2007. Improving Class Firewall Regression Test Selection by Removing the Class Firewall. International Journal of Software Engineering and Knowledge Engineering 17, 3 (2007), 359–378.
[76]
Peter Smith. 2011. Software Build Systems: Principles and Experience. Addison-Wesley Professional.
[77]
R. Spillane, R. Sears, C. Yalamanchili, S. Gaikwad, M. Chinni, and E. Zadok. 2009. Story Book: An Efficient Extensible Provenance Framework. In Workshop on on Theory and Practice of Provenance. 11:1–11:10.
[78]
Amitabh Srivastava and Jay Thiagarajan. 2002. Effectively Prioritizing Tests in Development Environment. In International Symposium on Software Testing and Analysis. 97–106.
[79]
Roman Suvorov, Meiyappan Nagappan, Ahmed E. Hassan, Ying Zou, and Bram Adams. 2012. An empirical study of build system migrations in practice: Case studies on KDE and the Linux kernel. In International Conference on Software Maintenance. 160–169.
[80]
Testing at the speed and scale of Google. http://google-engtools.blogspot.com/ 2011/06/testing-at-speed-and-scale-of-google.html.
[81]
TotT: Avoiding Flakey Tests. http://googletesting.blogspot.com/2008/04/ tott-avoiding-flakey-tests.html.
[82]
Mohsen Vakilian, Raluca Sauciuc, J. David Morgenthaler, and Vahab Mirrokni. 2015. Automated Decomposition of Build Targets. In International Conference on Software Engineering. 123–133.
[83]
Marko Vasic, Zuhair Parvez, Aleksandar Milicevic, and Milos Gligoric. 2017. File-Level vs. Module-Level Regression Test Selection for .NET. In Symposium on the Foundations of Software Engineering, Industry Track. TO APPEAR.
[84]
Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In International Symposium on Foundations of Software Engineering. 805–816.
[85]
David Willmor and Suzanne M. Embury. 2005. A Safe Regression Test Selection Technique for Database Driven Applications. In International Conference on Software Maintenance. 421–430.
[86]
Shin Yoo and Mark Harman. 2012. Regression Testing Minimization, Selection and Prioritization: A Survey. Journal of Software Testing, Verification and Reliability 22, 2 (2012), 67–120.
[87]
Lingming Zhang, Miryung Kim, and Sarfraz Khurshid. 2011. Localizing failureinducing program edits based on spectrum information. In International Conference on Software Maintenance. 23–32.
[88]
Sai Zhang, Darioush Jalali, Jochen Wuttke, Kivanç Muslu, Wing Lam, Michael D. Ernst, and David Notkin. 2014. Empirically revisiting the test independence assumption. In International Symposium on Software Testing and Analysis. 385– 396.

Cited By

View all
  • (2024)Hybrid Regression Test Selection by Integrating File and Method DependencesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695525(1557-1569)Online publication date: 27-Oct-2024
  • (2024)Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical AnalysisIEEE Transactions on Software Engineering10.1109/TSE.2024.341819150:8(2005-2020)Online publication date: Aug-2024
  • (2024)Impact of JVM Configurations on Test Runtime2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00032(249-261)Online publication date: 6-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering
August 2017
1073 pages
ISBN:9781450351058
DOI:10.1145/3106237
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Regression test selection
  2. language-agnostic

Qualifiers

  • Research-article

Conference

ESEC/FSE'17
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)0
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Hybrid Regression Test Selection by Integrating File and Method DependencesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695525(1557-1569)Online publication date: 27-Oct-2024
  • (2024)Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical AnalysisIEEE Transactions on Software Engineering10.1109/TSE.2024.341819150:8(2005-2020)Online publication date: Aug-2024
  • (2024)Impact of JVM Configurations on Test Runtime2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00032(249-261)Online publication date: 6-Oct-2024
  • (2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023
  • (2023)DIRTS: Dependency Injection Aware Regression Test Selection2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00046(422-432)Online publication date: Apr-2023
  • (2023)Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00044(398-409)Online publication date: Apr-2023
  • (2023)BinaryRTS: Cross-language Regression Test Selection for C++ Binaries in CI2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00038(327-338)Online publication date: Apr-2023
  • (2023)Test Selection for Unified Regression Testing2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00145(1687-1699)Online publication date: May-2023
  • (2023)An Approach for Test Impact Analysis on the Integration Level in Java ProgramsProceedings of Eighth International Congress on Information and Communication Technology10.1007/978-981-99-3091-3_14(171-188)Online publication date: 30-Jul-2023
  • (2022)Probe-based syscall tracing for efficient and practical file-level test tracesProceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test10.1145/3524481.3527239(126-137)Online publication date: 17-May-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media