research-article

Detecting, isolating, and enforcing dependencies among and within test cases

Author:

Jonathan BellAuthors Info & Claims

FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 799 - 802

https://doi.org/10.1145/2635868.2666597

Published: 11 November 2014 Publication History

Abstract

Testing stateful applications is challenging, as it can be difficult to identify hidden dependencies on program state. These dependencies may manifest between several test cases, or simply within a single test case. When it's left to developers to document, understand, and respond to these dependencies, a mistake can result in unexpected and invalid test results. Although current testing infrastructure does not currently leverage state dependency information, we argue that it could, and that by doing so testing can be improved. Our results thus far show that by recovering dependencies between test cases and modifying the popular testing framework, JUnit, to utilize this information, we can optimize the testing process, reducing time needed to run tests by 62% on average. Our ongoing work is to apply similar analyses to improve existing state of the art test suite prioritization techniques and state of the art test case generation techniques.

References

[1]

J. Bell and G. Kaiser. Phosphor: Illuminating Dynamic Data Flow in the JVM. OOPSLA ’14 (To Appear).

Digital Library

[2]

J. Bell and G. Kaiser. Unit Test Virtualization with VMVM. ICSE ’14.

Digital Library

[3]

J. Bell and G. Kaiser. VMVM: Unit Test Virtualization for Java (Formal Tool Demonstration).

[4]

J. Bell, N. Sarda, and G. Kaiser. Chronicler: Lightweight recording to reproduce field failures. ICSE ’13.

Digital Library

[5]

J. Bell, S. Sheth, and G. Kaiser. Secret ninja testing with halo software engineering. In Proc. of the 4th Int’l Workshop on Social Software Engineering, 2011.

Digital Library

[6]

J. Bell, S. Sheth, and G. Kaiser. A large-scale, longitudinal study of user profiles in world of warcraft. In Proc. of the 5th Int’l Workshop on Web Intelligence and Communities, 2013.

Digital Library

[7]

U. Buy, A. Orso, and M. Pezze. Automated testing of classes. ISSTA ’00.

Digital Library

[8]

C. Csallner and Y. Smaragdakis. Jcrasher: an automatic robustness tester for java. Software: Practice and Experience, 34(11):1025–1050, 2004.

Digital Library

[9]

M. J. Harrold and G. Rothermel. Performing data flow testing on classes. SIGSOFT ’94.

Digital Library

[10]

K. Inkumsah and T. Xie. Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution. ASE ’08.

Digital Library

[11]

H. Jaygarl, S. Kim, T. Xie, and C. K. Chang. Ocat: object capture-based automated testing. ISSTA ’10.

Digital Library

[12]

K. Mu¸slu, B. Soran, and J. Wuttke. Finding bugs by isolating unit tests. ESEC/FSE ’11.

[13]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. ICSE ’07.

Digital Library

[14]

G. Rothermel and M. J. Harrold. Analyzing regression test selection techniques. IEEE TSE, 1996.

Digital Library

[15]

S. Sheth, J. Bell, and G. Kaiser. HALO (Highly Addictive, sociaLly Optimized) Software Engineering. In Proc. of the 1st Int’l Workshop on Games and Software Engineering, 2011.

Digital Library

[16]

S. Thummalapenta, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. Mseqgen: Object-oriented unit-test generation via mining source code. ESEC/FSE ’09.

Digital Library

[17]

P. Tonella. Evolutionary testing of classes. ISSTA ’04.

Digital Library

[18]

M. Vivanti, A. Mis, A. Gorla, and G. Fraser. Search-based data-flow test generation. ISSRE ’13.

[19]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability, 22(2):67–120, Mar. 2012.

[20]

S. Zhang, D. Jalali, J. Wuttke, K. Muslu, M. Ernst, and D. Notkin. Empirically revisiting the test independence assumption. ISSTA ’14.

Digital Library

[21]

S. Zhang, D. Saff, Y. Bu, and M. D. Ernst. Combined static and dynamic automated test generation. ISSTA ’11.

Digital Library

Cited By

Wang HYi PParladorio JLam WMarinov DXie T(2024)Hierarchy-Aware Regression Test Prioritization2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00041(343-354)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00041
Chang ZLi MWang JWang QLi SRoychoudhury ACadar CKim M(2022)Putting them under microscope: a fine-grained approach for detecting redundant test cases in natural languageProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549089(1161-1172)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549089
Wei AYi PLi ZXie TMarinov DLam WDwyer MDamian DZeller A(2022)Preempting flaky tests via non-idempotent-outcome testsProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510170(1730-1742)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510170
Show More Cited By

Index Terms

Detecting, isolating, and enforcing dependencies among and within test cases
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...
Repairing test dependence
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

In a test suite, all the tests should be independent: no test should affect another test's result, and running the tests in any order should yield the same test results. The assumption of such test independence is important so that tests behave ...
Automated coverage calculation and test case generation
SAICSIT '12: Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference

This article describes the use of symbolic execution, a formal method of static analysis, to calculate code coverage of a program's existing JUnit test suites. Code coverage is measured with respect to a number of test adequacy criteria, including ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2014

856 pages

ISBN:9781450330565

DOI:10.1145/2635868

General Chair:
Shing-Chi Cheung
Hong Kong University of Science and Technology, China
,
Program Chairs:
Alessandro Orso
Georgia Institute of Technology, USA
,
Margaret-Anne Storey
University of Victoria, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SIGSOFT/FSE'14

Sponsor:

SIGSOFT

SIGSOFT/FSE'14: 22nd ACM SIGSOFT Symposium on the Foundations of Software Engineering

November 16 - 21, 2014

Hong Kong, China

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
204
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang HYi PParladorio JLam WMarinov DXie T(2024)Hierarchy-Aware Regression Test Prioritization2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00041(343-354)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00041
Chang ZLi MWang JWang QLi SRoychoudhury ACadar CKim M(2022)Putting them under microscope: a fine-grained approach for detecting redundant test cases in natural languageProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549089(1161-1172)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549089
Wei AYi PLi ZXie TMarinov DLam WDwyer MDamian DZeller A(2022)Preempting flaky tests via non-idempotent-outcome testsProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510170(1730-1742)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510170
Parry OKapfhammer GHilton MMcMinn P(2021)A Survey of Flaky TestsACM Transactions on Software Engineering and Methodology10.1145/347610531:1(1-74)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3476105
Lam WWinter SWei AXie TMarinov DBell J(2020)A large-scale longitudinal study of flaky testsProceedings of the ACM on Programming Languages10.1145/34282704:OOPSLA(1-29)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3428270
Yucesan OOzkil A(2020)Time Complexity Comparison of Stopping at First Failure and Completely Running the TestJournal of Electronic Testing10.1007/s10836-020-05884-336:3(409-417)Online publication date: 13-Jun-2020
https://doi.org/10.1007/s10836-020-05884-3
Ashouri M(2019)Practical Dynamic Taint Tracking for Exploiting Input Sanitization Error in Java ApplicationsInformation Security and Privacy10.1007/978-3-030-21548-4_27(494-513)Online publication date: 30-May-2019
https://doi.org/10.1007/978-3-030-21548-4_27
Guarnieri MTsankov PBuchs TTorabi Dashti MBasin DBultan TSen K(2017)Test execution checkpointing for web applicationsProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092710(203-214)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092710
Tahvili SBohlin MSaadatmand MLarsson SAfzal WSundmark D(2016)Cost-Benefit Analysis of Using Dependency Knowledge at Integration TestingProduct-Focused Software Process Improvement10.1007/978-3-319-49094-6_17(268-284)Online publication date: 6-Nov-2016
https://doi.org/10.1007/978-3-319-49094-6_17
Marroquin AGonzalez DMaag SNadimi ECerny TKim SWang W(2015)A novel distributed testing approach based on test cases dependencies for communication protocolsProceedings of the 2015 Conference on research in adaptive and convergent systems10.1145/2811411.2811554(497-504)Online publication date: 9-Oct-2015
https://dl.acm.org/doi/10.1145/2811411.2811554
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten