skip to main content
10.1145/3236024.3275527acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Regression test selection for TizenRT

Published: 26 October 2018 Publication History

Abstract

Regression testing - running tests after code modifications - is widely practiced in industry, including at Samsung. Regression Test Selection (RTS) optimizes regression testing by skipping tests that are not affected by recent code changes. Recent work has developed robust RTS tools, which mostly target managed languages, e.g., Java and C#, and thus are not applicable to large C projects, e.g., TizenRT, a lightweight RTOS-based platform.
We present Selfection, an RTS tool for projects written in C; we discuss the key challenges to develop Selfection and our design decisions. Selfection uses the objdump and readelf tools to statically build a dependency graph of functions from binaries and detect modified code elements. We integrated Selfection in TizenRT and evaluated its benefits if tests are run in an emulator and on a supported hardware platform (ARTIK 053). We used the latest 150 revisions of TizenRT available on GitHub. We measured the benefits of Selfection as the reduction in the number of tests and reduction in test execution time over running all tests at each revision (i.e., RetestAll). Our results show that Selfection can reduce, on average, the number of tests to 4.95% and end-to-end execution time to 7.04% when tests are executed in the emulator, and to 5.74% and 26.82% when tests are executed on the actual hardware. Our results also show that the time taken to maintain the dependency graph and detect modified functions is negligible.

References

[1]
Testing at the speed and scale of Google. http://google-engtools.blogspot.com/ 2011/06/testing-at-speed-and-scale-of-google.html.
[2]
Tools for Continuous Integration at Google Scale. http://www.youtube.com/ watch?v=b52aXZ2yi08.
[3]
Artik Home Page. Samsung ARTIK IoT Platform. https://www.artik.io/modules/ artik-05x.
[4]
Bazel Home Page. Build and test software of any size, quickly and reliably. https://bazel.build.
[5]
Jonathan Bell, Gail Kaiser, Eric Melski, and Mohan Dattatreya. 2015. Efficient Dependency Detection for Safe Java Test Acceleration. In FSE. 770–781.
[6]
Swarnendu Biswas, Rajib Mall, Manoranjan Satpathy, and Srihari Sukumaran. 2011. Regression Test Selection Techniques: A Survey. Informatica (Slovenia) 35, 3 (2011), 289–321.
[7]
Vincent Blondeau, Anne Etien, Nicolas Anquetil, Sylvain Cresson, Pascal Croisy, and Stéphane Ducasse. 2017. What are the Testing Habits of Developers? A Case Study in a Large IT Company. In ICSME.
[8]
Ahmet Celik, Alex Knaust, Aleksandar Milicevic, and Milos Gligoric. 2016. Build System with Lazy Retrieval for Java Projects. In FSE. 643–654.
[9]
Yih-Farn Chen, David S. Rosenblum, and Kiem-Phong Vo. 1994. TestTube: A System for Selective Regression Testing. In ICSE. 211–220.
[10]
Sebastian Elbaum, Gregg Rothermel, and John Penix. 2014. Techniques for Improving Regression Testing in Continuous Integration Development Environments. In FSE. 235–245.
[11]
Emelie Engström and Per Runeson. 2010. A Qualitative Survey of Regression Testing Practices. In PROFES. 3–16.
[12]
Emelie Engström, Per Runeson, and Mats Skoglund. 2010. A Systematic Review on Regression Test Selection Techniques. IST 52, 1 (2010), 14–30.
[13]
Hamed Esfahani, Jonas Fietz, Qi Ke, Alexei Kolomiets, Erica Lan, Erik Mavrinac, Wolfram Schulte, Newton Sanches, and Srikanth Kandula. 2016. CloudBuild: Microsoft’s Distributed and Caching Build Service. In ICSE, SEIP. 11–20.
[14]
Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Practical Regression Test Selection with Dynamic File Dependencies. In ISSTA. 211–222.
[15]
Milos Gligoric, Stas Negara, Owolabi Legunsen, and Darko Marinov. 2014. An Empirical Evaluation and Comparison of Manual and Automated Test Selection. In ASE. 361–372.
[16]
Alex Gyori, August Shi, Farah Hariri, and Darko Marinov. 2015. Reliable testing: detecting state-polluting tests to prevent test dependency. In ISSTA. 223–233.
[17]
Kim Herzig, Michaela Greiler, Jacek Czerwonka, and Brendan Murphy. 2015. The Art of Testing Less without Sacrificing Quality. In ICSE. 483–493.
[18]
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in Continuous Integration: Assurance, Security, and Flexibility. In FSE. 197–207.
[19]
Owolabi Legunsen, Farah Hariri, August Shi, Yafeng Lu, Lingming Zhang, and Darko Marinov. 2016. An Extensive Study of Static Regression Test Selection in Modern Software Evolution. In FSE. 583–594.
[20]
Qemu Home Page. QEMU - the FAST! processor emulator. https://www.qemu.org.
[21]
Xiaoxia Ren, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia Chesley. 2004. Chianti: A tool for change impact analysis of Java programs. In OOPSLA. 432–448.
[22]
Gregg Rothermel and Mary Jean Harrold. 1993. A safe, efficient algorithm for regression test selection. In ICSM. 358–367.
[23]
Gregg Rothermel and Mary Jean Harrold. 1996. Analyzing Regression Test Selection Techniques. TSE 22, 8 (1996), 529–551.
[24]
Gregg Rothermel and Mary Jean Harrold. 1997. A safe, efficient regression test selection technique. TOSEM 6, 2 (1997), 173–210.
[25]
Mats Skoglund and Per Runeson. 2007. Improving Class Firewall Regression Test Selection by Removing the Class Firewall. IJSEKE 17, 3 (2007), 359–378.
[26]
Amitabh Srivastava and Jay Thiagarajan. 2002. Effectively Prioritizing Tests in Development Environment. In ISSTA. 97–106.
[27]
TizenRT Home Page. TizenRT - Lightweight RTOS-based platform for low-end IoT devices. https://github.com/Samsung/TizenRT.
[28]
TizenRT Pull1368. Should deinitialize database after removing relations. https: //github.com/Samsung/TizenRT/pull/1368.
[29]
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2017. There and Back Again: Can you Compile that Snapshot? JSEP (2017).
[30]
Marko Vasic, Zuhair Parvez, Aleksandar Milicevic, and Milos Gligoric. 2017. File-Level vs. Module-Level Regression Test Selection for .NET. In FSE, industry track. 848–853.
[31]
Shin Yoo and Mark Harman. 2012. Regression Testing Minimization, Selection and Prioritization: A Survey. STVR 22, 2 (2012), 67–120.

Cited By

View all
  • (2024)When Automated Program Repair Meets Regression Testing—An Extensive Study on Two Million PatchesACM Transactions on Software Engineering and Methodology10.1145/367245033:7(1-23)Online publication date: 13-Jun-2024
  • (2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023
  • (2023)An information retrieval-based regression test selection techniqueIran Journal of Computer Science10.1007/s42044-023-00145-w6:4(365-373)Online publication date: 15-May-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
October 2018
987 pages
ISBN:9781450355735
DOI:10.1145/3236024
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Regression test selection
  2. TizenRT
  3. static dependency analysis

Qualifiers

  • Research-article

Conference

ESEC/FSE '18
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)11
  • Downloads (Last 6 weeks)1
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)When Automated Program Repair Meets Regression Testing—An Extensive Study on Two Million PatchesACM Transactions on Software Engineering and Methodology10.1145/367245033:7(1-23)Online publication date: 13-Jun-2024
  • (2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023
  • (2023)An information retrieval-based regression test selection techniqueIran Journal of Computer Science10.1007/s42044-023-00145-w6:4(365-373)Online publication date: 15-May-2023
  • (2019)Reflection-aware static regression test selectionProceedings of the ACM on Programming Languages10.1145/33606133:OOPSLA(1-29)Online publication date: 10-Oct-2019
  • (2019)QADroid: regression event selection for Android applicationsProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3293882.3330550(66-77)Online publication date: 10-Jul-2019
  • (2019) MahtabJournal of Systems and Software10.1016/j.jss.2019.110403158:COnline publication date: 1-Dec-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media