skip to main content
10.1145/3545258.3545280acmotherconferencesArticle/Chapter ViewAbstractPublication PagesinternetwareConference Proceedingsconference-collections
research-article

Randoop-TSR: Random-based Test Generator with Test Suite Reduction

Published: 15 September 2022 Publication History

Abstract

Software testing plays a very important role in the software development process. Automated test generation tools increase the effectiveness and efficiency of software testing, and alleviate the problem of low efficiency caused by writing hand-crafted test cases. However, different test case generation methods vary in the size, code coverage, and fault detection capacity of the automatically-produced test suites. Automated test case generation tool based on random testing, Randoop as a representative, randomly and incrementally generates a large number of method sequences, which gives various possible combinations of calling methods, but the size of the test suite is not proportional to test quality. Therefore, there exists a lot of redundancy in the test cases. This paper proposes Randoop-TSR, an approach for identifying and eliminating redundant test cases on the basis of Randoop to improve the process of test generation. Our approach adopts three strategies to realize the removal of redundancy, namely: (i) similarity-based input sequence selection; (ii) redundant and duplicate assert statements elimination based on test smell detection; (iii) redundant test cases elimination without breaking test requirements (i.e., code coverage and mutation score). Randoop-TSR can eliminate redundancy effectively, and greatly reduce the size of test suites and execution time. Furthermore, our approach improves the efficiency and understandability of test cases while retaining code coverage and mutation score.

Supplementary Material

Presentation slides (PPT-Randoop-TSR.pdf)

References

[1]
2022. Randoop Manual. https://randoop.github.io/randoop/manual/.
[2]
M Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 263–272.
[3]
Saswat Anand, Edmund K Burke, Tsong Yueh Chen, John Clark, Myra B Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil Mcminn, Antonia Bertolino, 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86, 8 (2013), 1978–2001.
[4]
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? an empirical study. Empirical Software Engineering 20, 4 (2015), 1052–1094.
[5]
Pietro Braione and Giovanni Denaro. 2019. SUSHI and TARDIS at the SBST2019 Tool Competition. In 2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST). IEEE, 25–28.
[6]
Pietro Braione, Giovanni Denaro, Andrea Mattavelli, and Mauro Pezzè. 2017. Combining symbolic execution and search-based testing for programs with complex heap inputs. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 90–101.
[7]
Junjie Chen, Yanwei Bai, Dan Hao, Lingming Zhang, Lu Zhang, and Bing Xie. 2017. How do assertions impact coverage-based test-suite reduction?. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 418–423.
[8]
Tsong Yueh Chen, Fei-Ching Kuo, Dave Towey, and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58, 5 (2015), 1–9.
[9]
Tsong Yueh Chen and Man Fai Lau. 1998. A new heuristic for test suite reduction. Information and Software Technology 40, 5-6 (1998), 347–354.
[10]
Tsong Yueh Chen, Hing Leung, and IK Mak. 2004. Adaptive random testing. In Annual Asian Computing Science Conference. Springer, 320–329.
[11]
Nour Chetouane, Franz Wotawa, Hermann Felbinger, and Mihai Nica. 2020. On Using k-means Clustering for Test Suite Reduction. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 380–385.
[12]
Koen Claessen and John Hughes. 2011. QuickCheck: a lightweight tool for random testing of Haskell programs. Acm sigplan notices 46, 4 (2011), 53–64.
[13]
Graham Cormode, Howard Karloff, and Anthony Wirth. 2010. Set cover algorithms for very large datasets. In Proceedings of the 19th ACM international conference on Information and knowledge management. 479–488.
[14]
Carmen Coviello, Simone Romano, Giuseppe Scanniello, Alessandro Marchetto, Giuliano Antoniol, and Anna Corazza. 2018. Clustering support for inadequate test suite reduction. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 95–105.
[15]
Carmen Coviello, Simone Romano, Giuseppe Scanniello, Alessandro Marchetto, Anna Corazza, and Giuliano Antoniol. 2020. Adequate vs. inadequate test suite reduction approaches. Information and Software Technology 119 (2020), 106224.
[16]
Emilio Cruciani, Breno Miranda, Roberto Verdecchia, and Antonia Bertolino. 2019. Scalable approaches for test suite reduction. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 419–429.
[17]
Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: an automatic robustness tester for Java. Software: Practice and Experience 34, 11 (2004), 1025–1050.
[18]
Marko Dimjaševic and Zvonimir Rakamaric. 2013. JPF-Doop: Combining concolic and random testing for Java. Collections (org. apache. commons. collections) 422, 3894(2013), 58470.
[19]
Sebastian Elbaum, Alexey G Malishevsky, and Gregg Rothermel. 2002. Test case prioritization: A family of empirical studies. IEEE transactions on software engineering 28, 2 (2002), 159–182.
[20]
Leonardo Fernandes, Márcio Ribeiro, Luiz Carvalho, Rohit Gheyi, Melina Mongiovi, André Santos, Ana Cavalcanti, Fabiano Ferrari, and José Carlos Maldonado. 2017. Avoiding useless mutants. In Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 187–198.
[21]
Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419.
[22]
Jingyao Geng, Zheng Li, Ruilian Zhao, and Junxia Guo. 2016. Search based test suite minimization for fault detection and localization: A co-driven method. In International Symposium on Search Based Software Engineering. Springer, 34–48.
[23]
Giovanni Grano, Simone Scalabrino, Harald C Gall, and Rocco Oliveto. 2018. An empirical investigation on the readability of manual and generated test cases. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC). IEEE, 348–3483.
[24]
M Jean Harrold, Rajiv Gupta, and Mary Lou Soffa. 1993. A methodology for controlling the size of a test suite. ACM Transactions on Software Engineering and Methodology (TOSEM) 2, 3(1993), 270–285.
[25]
Rubing Huang, Weifeng Sun, Yinyin Xu, Haibo Chen, Dave Towey, and Xin Xia. 2019. A survey on adaptive random testing. IEEE Transactions on Software Engineering(2019).
[26]
Wei Huang, Ana Milanova, Werner Dietl, and Michael D Ernst. 2012. ReIm & ReImInfer: Checking and inference of reference immutability and method purity. ACM SIGPLAN Notices 47, 10 (2012), 879–896.
[27]
Hojun Jaygarl, Kai-Shin Lu, and Carl K Chang. 2010. GenRed: A tool for generating and reducing object-oriented test cases. In 2010 IEEE 34th Annual Computer Software and Applications Conference. IEEE, 127–136.
[28]
Rafaqut Kazmi, Dayang NA Jawawi, Radziah Mohamad, and Imran Ghani. 2017. Effective regression test case selection: A systematic literature review. ACM Computing Surveys (CSUR) 50, 2 (2017), 1–32.
[29]
Saif Ur Rehman Khan, Sai Peck Lee, Nadeem Javaid, and Wadood Abdul. 2018. A systematic review on test suite reduction: Approaches, experiment’s quality evaluation, and guidelines. IEEE Access 6(2018), 11816–11841.
[30]
Chu-Ti Lin, Kai-Wei Tang, Jiun-Shiang Wang, and Gregory M Kapfhammer. 2017. Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity. Science of Computer Programming 150 (2017), 1–25.
[31]
Lei Ma, Cyrille Artho, Cheng Zhang, Hiroyuki Sato, Johannes Gmeiner, and Rudolf Ramler. 2015. Grt: Program-analysis-guided random testing (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 212–223.
[32]
Yu-Seung Ma and Jeff Offutt. 2005. Description of class mutation mutation operators for java. Electronics and Telecommunications Research Institute, Korea (2005).
[33]
Yu-Seung Ma and Jeff Offutt. 2016. Description of mujava’s method-level mutation operators. Update (2016).
[34]
Yu-Seung Ma, Jeff Offutt, and Yong-Rae Kwon. 2006. MuJava: a mutation system for Java. In Proceedings of the 28th international conference on Software engineering. 827–830.
[35]
François Margot. 2010. Symmetry in integer linear programming. 50 Years of Integer Programming 1958-2008(2010), 647–686.
[36]
Misael Mongiovì, Andrea Fornaia, and Emiliano Tramontana. 2020. REDUNET: reducing test suites by integrating set cover and network-based optimization. Applied Network Science 5, 1 (2020), 1–21.
[37]
İbrahim Muter, Ş İlker Birbil, and Güvenç Şahin. 2010. Combination of metaheuristic and exact algorithms for solving set covering-type optimization problems. INFORMS Journal on Computing 22, 4 (2010), 603–619.
[38]
Carlos Pacheco and Michael D Ernst. 2007. Randoop: feedback-directed random testing for Java. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion. 815–816.
[39]
Carlos Pacheco, Shuvendu K Lahiri, Michael D Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE’07). IEEE, 75–84.
[40]
Fabio Palomba, Dario Di Nucci, Annibale Panichella, Rocco Oliveto, and Andrea De Lucia. 2016. On the diffusion of test smells in automatically generated test code: An empirical study. In 2016 IEEE/ACM 9th International Workshop on Search-Based Software Testing (SBST). IEEE, 5–14.
[41]
Pablo Ponzio, Valeria Bengolea, Simón Gutiérrez Brida, Gastón Scilingo, Nazareno Aguirre, and Marcelo Frias. 2018. On the effect of object redundancy elimination in randomly testing collection classes. In 2018 IEEE/ACM 11th International Workshop on Search-Based Software Testing (SBST). IEEE, 67–70.
[42]
IS Wishnu B Prasetya. 2013. T3, a combinator-based random testing tool for java: benchmarking. In International Workshop on Future Internet Testing. Springer, 101–110.
[43]
Brian Robinson, Michael D Ernst, Jeff H Perkins, Vinay Augustine, and Nuo Li. 2011. Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs. In 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011). IEEE, 23–32.
[44]
José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2015. Automated unit test generation during software development: A controlled experiment and think-aloud observations. In Proceedings of the 2015 international symposium on software testing and analysis. 338–349.
[45]
Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 2001. Prioritizing test cases for regression testing. IEEE Transactions on software engineering 27, 10 (2001), 929–948.
[46]
Abdelilah Sakti, Gilles Pesant, and Yann-Gaël Guéhéneuc. 2014. Instance generator and problem representation to improve object oriented code coverage. IEEE Transactions on Software Engineering 41, 3 (2014), 294–313.
[47]
Nastassia Smeets and Anthony JH Simons. 2011. Automated Unit Testing with Randoop, JWalk and μJava versus Manual JUnit Testing. Technical Report. Research report, Department of Computer Science, University of Sheffield ….
[48]
Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001). Citeseer, 92–95.
[49]
Belma Yelbay, Ş İlker Birbil, and Kerem Bülbül. 2014. The set covering problem revisited: an empirical study of the value of dual information. Journal of Industrial and Management Optimization (2014).
[50]
Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software testing, verification and reliability 22, 2(2012), 67–120.
[51]
Yanbing Yu, James Jones, and Mary Jean Harrold. 2008. An empirical study of the effects of test-suite reduction on fault localization. In 2008 ACM/IEEE 30th International Conference on Software Engineering. IEEE, 201–210.
[52]
Yucheng Zhang and Ali Mesbah. 2015. Assertions are strongly correlated with test suite effectiveness. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 214–224.

Cited By

View all
  • (2024)Element and Event-Based Test Suite Reduction for Android Test Suites Generated by Reinforcement Learningundefined10.12794/metadc2356236Online publication date: Jul-2024
  • (2024)A Review of the Applications of Heuristic Algorithms in Test Case Generation Problem2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C63300.2024.00114(856-865)Online publication date: 1-Jul-2024
  • (2024)Element-Based Test Suite Reduction for SARSA-Generated Test SuitesITNG 2024: 21st International Conference on Information Technology-New Generations10.1007/978-3-031-56599-1_50(387-395)Online publication date: 11-Mar-2024

Index Terms

  1. Randoop-TSR: Random-based Test Generator with Test Suite Reduction
      Index terms have been assigned to the content through auto-classification.

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      Internetware '22: Proceedings of the 13th Asia-Pacific Symposium on Internetware
      June 2022
      291 pages
      ISBN:9781450397803
      DOI:10.1145/3545258
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 15 September 2022

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Automated test generation tool
      2. Random Testing
      3. Test Suite Reduction

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      • the National Natural Science Foundation of China

      Conference

      Internetware 2022

      Acceptance Rates

      Overall Acceptance Rate 55 of 111 submissions, 50%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)32
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 15 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Element and Event-Based Test Suite Reduction for Android Test Suites Generated by Reinforcement Learningundefined10.12794/metadc2356236Online publication date: Jul-2024
      • (2024)A Review of the Applications of Heuristic Algorithms in Test Case Generation Problem2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C63300.2024.00114(856-865)Online publication date: 1-Jul-2024
      • (2024)Element-Based Test Suite Reduction for SARSA-Generated Test SuitesITNG 2024: 21st International Conference on Information Technology-New Generations10.1007/978-3-031-56599-1_50(387-395)Online publication date: 11-Mar-2024

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media