research-article

Multi-objective Integer Programming Approaches for Solving the Multi-criteria Test-suite Minimization Problem: Towards Sound and Complete Solutions of a Particular Search-based Software-engineering Problem

Authors:
Yinxing Xue

School of Computer Science and Technology, University of Science and Technology of China, China

School of Computer Science and Technology, University of Science and Technology of China, China
View Profile

,
Yan-Fu Li

Department of Industrial Engineering, Tsinghua University, China

Department of Industrial Engineering, Tsinghua University, China

0000-0001-5755-7115
View Profile

ACM Transactions on Software Engineering and Methodology Volume 29 Issue 3Article No.: 20pp 1–50https://doi.org/10.1145/3392031

Published:01 June 2020Publication History

ACM Transactions on Software Engineering and Methodology

Abstract

Test-suite minimization is one key technique for optimizing the software testing process. Due to the need to balance multiple factors, multi-criteria test-suite minimization (MCTSM) becomes a popular research topic in the recent decade. The MCTSM problem is typically modeled as integer linear programming (ILP) problem and solved with weighted-sum single objective approach. However, there is no existing approach that can generate sound (i.e., being Pareto-optimal) and complete (i.e., covering the entire Pareto front) Pareto-optimal solution set, to the knowledge of the authors. In this work, we first prove that the ILP formulation can accurately model the MCTSM problem and then propose the multi-objective integer programming (MOIP) approaches to solve it. We apply our MOIP approaches on three specific MCTSM problems and compare the results with those of the cutting-edge methods, namely, NonlinearFormulation_LinearSolver (NF_LS) and two Multi-Objective Evolutionary Algorithms (MOEAs). The results show that our MOIP approaches can always find sound and complete solutions on five subject programs, using similar or significantly less time than NF_LS and two MOEAs do. The current experimental results are quite promising, and our approaches have the potential to be applied for other similar search-based software engineering problems.

References

Radio Technical Commission for Aeronautics (RTCA). 1992. DO-178B: Software Considerations in Airborne Systems and Equipment Certification. https://en.wikipedia.org/wiki/DO-178B.Google Scholar
Sven Apel and Christian Kästner. 2009. An overview of feature-oriented software development. J. Obj. Technol. 8, 5 (2009), 49--84. DOI:https://doi.org/10.5381/jot.2009.8.5.c5Google ScholarCross Ref
Stephan Arlt, Andreas Podelski, and Martin Wehrle. 2014. Reducing GUI test suites via program slicing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’14). 270--281. DOI:https://doi.org/10.1145/2610384.2610391Google ScholarDigital Library
Aitor Arrieta, Shuai Wang, Urtzi Markiegi, Ainhoa Arruabarrena, Leire Etxeberria, and Goiuria Sagardui. 2019. Pareto efficient multi-objective black-box test case selection for simulation-based testing. Inf. Softw. Technol. 114 (2019), 137--154. DOI:https://doi.org/10.1016/j.infsof.2019.06.009Google ScholarDigital Library
Hauke Baller, Sascha Lity, Malte Lochau, and Ina Schaefer. 2014. Multi-objective test-suite optimization for incremental product family testing. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation ICST’14). 303--312. DOI:https://doi.org/10.1109/ICST.2014.43Google ScholarDigital Library
Don S. Batory. 2005. Feature models, grammars, and propositional formulas. In Proceedings of the 9th International Conference on Software Product Lines (SPLC’05). 7--20. DOI:https://doi.org/10.1007/11554844_3Google ScholarDigital Library
Benoit Baudry, Franck Fleurey, and Yves Le Traon. 2006. Improving test suites for efficient fault localization. In Proceedings of the 28th International Conference on Software Engineering (ICSE’06). 82--91. DOI:https://doi.org/10.1145/1134285.1134299Google ScholarDigital Library
Reyhaneh Jabbarvand Behrouz, Alireza Sadeghi, Hamid Bagheri, and Sam Malek. 2016. Energy-aware test-suite minimization for Android apps. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’16). ACM, 425--436.Google Scholar
Jennifer Black, Emanuel Melachrinoudis, and David R. Kaeli. 2004. Bi-criteria models for all-uses test suite reduction. In Proceedings of the 26th International Conference on Software Engineering (ICSE’04). 106--115. DOI:https://doi.org/10.1109/ICSE.2004.1317433Google ScholarCross Ref
Marcel Böhme and Abhik Roychoudhury. 2014. CoREBench: Studying complexity of regression errors. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’14). 105--115. DOI:https://doi.org/10.1145/2610384.2628058Google ScholarDigital Library
Prasad Bokil, Padmanabhan Krishnan, and R. Venkatesh. 2015. Achieving effective test suites for reactive systems using specification mining and test suite reduction techniques. ACM SIGSOFT Softw. Eng. Notes 40, 1 (2015), 1--8. DOI:https://doi.org/10.1145/2693208.2693226Google ScholarDigital Library
Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press, New York, NY.Google ScholarDigital Library
Lionel C. Briand, Yvan Labiche, Zaheer Bawar, and Nadia Traldi Spido. 2009. Using machine learning to refine category-partition test specifications and test suites. Inf. Softw. Technol. 51, 11 (2009), 1551--1564. DOI:https://doi.org/10.1016/j.infsof.2009.06.006Google ScholarDigital Library
Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI’08). 209--224. Retrieved from http://www.usenix.org/events/osdi08/tech/full_papers/cadar/cadar.pdf.Google ScholarDigital Library
Pär Carlshamre. 2002. Release planning in market-driven software product development: Provoking an understanding. Requir. Eng. 7, 3 (2002), 139--151.Google ScholarCross Ref
Ryan Carlson, Hyunsook Do, and Anne Denton. 2011. A clustering approach to improving test case prioritization: An industrial case study. In Proceedings of the IEEE 27th International Conference on Software Maintenance (ICSM’11). 382--391. DOI:https://doi.org/10.1109/ICSM.2011.6080805Google ScholarDigital Library
Tsong Yueh Chen and Man Fai Lau. 1996. Dividing strategies for the optimization of a test suite. Inf. Proc. Lett. 60, 3 (1996), 135--141. DOI:https://doi.org/10.1016/S0020-0190(96)00135-4Google ScholarDigital Library
Carlos A. Coello and Efrén Mezura-Montes. 2002. Constraint-handling in genetic algorithms through the use of dominance-based tournament selection. Adv. Eng. Inform. 16, 3 (2002), 193--203.Google ScholarCross Ref
Domenico Cotroneo, Roberto Pietrantuono, and Stefano Russo. 2013. A learning-based method for combining testing techniques. In Proceedings of the 35th International Conference on Software Engineering (ICSE’13). 142--151. DOI:https://doi.org/10.1109/ICSE.2013.6606560Google ScholarCross Ref
Emilio Cruciani, Breno Miranda, Roberto Verdecchia, and Antonia Bertolino. 2019. Scalable approaches for test suite reduction. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). 419--429. DOI:https://doi.org/10.1109/ICSE.2019.00055Google ScholarDigital Library
Tibor Csöndes, Sarolta Dibuz, and Balázs Kotnyek. 1999. Test suite reduction in conformance testing. Acta Cybern. 14 (1 1999), 229--238.Google Scholar
Dario Izzo and Francesco Biscani. 2019. Pagmo (C++) or Pygmo (Python): A scientific library for massively parallel optimization. Retrieved from https://esa.github.io/pagmo2/.Google Scholar
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 2 (2002), 182--197. DOI:https://doi.org/10.1109/4235.996017Google ScholarDigital Library
Orit Edelstein, Eitan Farchi, Evgeny Goldin, Yarden Nir, Gil Ratsaby, and Shmuel Ur. 2003. Framework for testing multi-threaded Java programs. Concurr. Comput. Pract. Exper. 15, 3–5 (2003), 485--499. DOI:https://doi.org/10.1002/cpe.654Google ScholarCross Ref
Sebastian Elbaum, Alexey G. Malishevsky, and Gregg Rothermel. 2002. Test case prioritization: A family of empirical studies. IEEE Trans. Softw. Eng. 28, 2 (Feb. 2002), 159--182. DOI:https://doi.org/10.1109/32.988497Google ScholarDigital Library
Emelie Engström, Per Runeson, and Mats Skoglund. 2010. A systematic review on regression test selection techniques. Inf. Softw. Technol. 52, 1 (2010), 14--30. DOI:https://doi.org/10.1016/j.infsof.2009.07.001Google ScholarDigital Library
Usman Farooq and Chiou Peng Lam. 2009. Evolving the quality of a model based test suite. In Proceedings of the 2nd International Conference on Software Testing Verification and Validation (ICST’09). 141--149. DOI:https://doi.org/10.1109/ICSTW.2009.27Google ScholarDigital Library
Michael R. Garey and David S. Johnson. 1990. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman 8 Co., New York, NY.Google ScholarDigital Library
GCC 2018. gcov—A Test Coverage Program. Retrieved from https://gcc.gnu.org/onlinedocs/gcc/Gcov.html.Google Scholar
Jingyao Geng, Zheng Li, Ruilian Zhao, and Junxia Guo. 2016. Search based test suite minimization for fault detection and localization: A co-driven method. In Proceedings of the 8th International Symposium on Search-based Software Engineering (SSBSE’16). 34--48. DOI:https://doi.org/10.1007/978-3-319-47106-8_3Google ScholarCross Ref
Qing Gu, Bao Tang, and Daoxu Chen. 2010. Optimal regression testing based on selective coverage of test requirements. In Proceedings of the IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA’10). 419--426. DOI:https://doi.org/10.1109/ISPA.2010.62Google ScholarDigital Library
Liucheng Guo, Jiangfang Yi, Liang Zhang, Xiaoyin Wang, and Dong Tong. 2011. CGA: Combining cluster analysis with genetic algorithm for regression suite reduction of microprocessors. In Proceedings of the IEEE International System-on-Chip Conference (SoCC’11). 207--212. DOI:https://doi.org/10.1109/SOCC.2011.6085105Google ScholarCross Ref
Brent Hailpern and Padmanabhan Santhanam. 2001. Software Debugging, Testing, and Verification. IBM Research Report RC22220 (W0110-083). IBM, Thomas J. Watson Research Center, NY.Google Scholar
Yacov Y. Haimes, Leon S. Lasdon, and David A. Wismer. 1971. On a bicriterion formulation of the problems of integrated system identification and system optimization. IEEE Trans. Syst. Man Cybern. SMC-1, 3 (July 1971), 296--297. DOI:https://doi.org/10.1109/TSMC.1971.4308298Google Scholar
Dan Hao, Lu Zhang, Xingxia Wu, Hong Mei, and Gregg Rothermel. 2012. On-demand test suite reduction. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). 738--748. DOI:https://doi.org/10.1109/ICSE.2012.6227144Google ScholarDigital Library
Mary Jean Harrold, Rajiv Gupta, and Mary Lou Soffa. 1993. A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2, 3 (1993), 270--285. DOI:https://doi.org/10.1145/152388.152391Google ScholarDigital Library
Christopher Henard, Mike Papadakis, Mark Harman, Yue Jia, and Yves Le Traon. 2016. Comparing white-box and black-box test prioritization. In Proceedings of the 38th International Conference on Software Engineering (ICSE’16). 523--534. DOI:https://doi.org/10.1145/2884781.2884791Google ScholarDigital Library
Mark Hennessy and James F. Power. 2005. An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE’05). 104--113. DOI:https://doi.org/10.1145/1101908.1101926Google Scholar
Hwa-You Hsu and Alessandro Orso. 2009. MINTS: A general framework and tool for supporting test-suite minimization. In Proceedings of the 31st International Conference on Software Engineering (ICSE’09). 419--429. DOI:https://doi.org/10.1109/ICSE.2009.5070541Google ScholarDigital Library
I. Griva, S. G. Nash, and A. Sofer. 2009. Linear and Nonlinear Optimization (2nd. ed.). Society for Industrial Mathematics.Google Scholar
IBM Inc. 2019. CPLEX Optimizer. Retrieved from https://www.ibm.com/analytics/cplex-optimizerGoogle Scholar
Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering (ICSE’14). 435--445. DOI:https://doi.org/10.1145/2568225.2568271Google ScholarDigital Library
Wei Jin. 2013. Reproducing and debugging field failures in house. In Proceedings of the 35th International Conference on Software Engineering (ICSE’13). 1441--1443. DOI:https://doi.org/10.1109/ICSE.2013.6606738Google ScholarCross Ref
Alireza Khalilian and Saeed Parsa. 2009. Bi-criteria test suite reduction by cluster analysis of execution profiles. In Proceedings of the Central and East European Conference on Software Engineering Techniques (CEE-SET’09). 243--256. DOI:https://doi.org/10.1007/978-3-642-28038-2_19Google Scholar
Saif Rehman Khan, Sai Peck Lee, Nadeem Javaid, and Wadood Abdul. 2018. A systematic review on test suite reduction: Approaches, experiment’s quality evaluation, and guidelines. IEEE Access 6 (2018), 11816--11841. DOI:https://doi.org/10.1109/ACCESS.2018.2809600Google ScholarCross Ref
Muhammad Khatibsyarbini, Mohd Adham Isa, Dayang N. A. Jawawi, and Rooster Tumeng. 2018. Test case prioritization approaches in regression testing: A systematic literature review. Inf. Softw. Technol. 93 (2018), 74--93.Google ScholarDigital Library
Adriaan Labuschagne, Laura Inozemtseva, and Reid Holmes. 2017. Measuring the cost of regression testing in practice: A study of Java projects using continuous integration. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). 821--830. DOI:https://doi.org/10.1145/3106237.3106288Google ScholarDigital Library
Remo Lachmann, Michael Felderer, Manuel Nieke, Sandro Schulze, Christoph Seidl, and Ina Schaefer. 2017. Multi-objective black-box test case selection for system testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’17). 1311--1318. DOI:https://doi.org/10.1145/3071178.3071189Google ScholarDigital Library
Andreas Leitner, Manuel Oriol, Andreas Zeller, Ilinca Ciupa, and Bertrand Meyer. 2007. Efficient unit test case minimization. In Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE’07). 417--420. DOI:https://doi.org/10.1145/1321631.1321698Google ScholarDigital Library
Hareton K. N. Leung and Lee White. 1991. A cost model to compare regression test strategies. In Proceedings of the International Conference on Software Maintenance. 201--208. DOI:https://doi.org/10.1109/ICSM.1991.160330Google Scholar
Chen Li, Marjan van den Akker, Sjaak Brinkkemper, and Guido Diepen. 2010. An integrated approach for requirement selection and scheduling in software release planning. Requir. Eng. 15, 4 (2010), 375--396. DOI:https://doi.org/10.1007/s00766-010-0104-xGoogle ScholarDigital Library
Ding Li, Yuchen Jin, Cagri Sahin, James Clause, and William G. J. Halfond. 2014. Integrated energy-directed test suite optimization. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’14). ACM, 339--350.Google Scholar
Yan-Fu Li and Enrico Zio. 2017. RAMS optimization principles. Handbook of Safety Principles. Wiley, 514--539.Google Scholar
Jun-Wei Lin and Chin-Yu Huang. 2009. Analysis of test suite reduction with enhanced tie-breaking techniques. Inf. Softw. Technol. 51, 4 (2009), 679--690. DOI:https://doi.org/10.1016/j.infsof.2008.11.004Google ScholarDigital Library
Jun-Wei Lin, Reyhaneh Jabbarvand, Joshua Garcia, and Sam Malek. 2018. Nemo: Multi-criteria test-suite minimization with integer nonlinear programming. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). 1039--1049. DOI:https://doi.org/10.1145/3180155.3180174Google ScholarDigital Library
Yue Liu, Kang Wang, Wang Wei, Bofeng Zhang, and Hailin Zhong. 2011. User-session-based test cases optimization method based on agglutinate hierarchy clustering. In Proceedings of the IEEE International Conference on Internet of Things and International Conference on Cyber, Physical, and Social Computing (iThings 8 CPSCom’11). 413--418. DOI:https://doi.org/10.1109/iThings/CPSCom.2011.135Google ScholarDigital Library
Henry B. Mann and Donald R. Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18, 1 (1947), 50--60.Google ScholarCross Ref
Alessandro Marchetto, Md. Mahfuzul Islam, M. Waseem Asghar, Angelo Susi, and Giuseppe Scanniello. 2016. A multi-objective technique to prioritize test cases. IEEE Trans. Softw. Eng. 42, 10 (2016), 918--940. DOI:https://doi.org/10.1109/TSE.2015.2510633Google ScholarDigital Library
Alessandro Marchetto, Md. Mahfuzul Islam, Angelo Susi, and Giuseppe Scanniello. 2013. A multi-objective technique for test suite reduction. In Proceedings of the 8th International Conference on Software Engineering Advances (ICSEA’13). 18--24.Google Scholar
Martina Marré and Antonia Bertolino. 2003. Using spanning sets for coverage testing. IEEE Trans. Softw. Eng. 29, 11 (2003), 974--984. DOI:https://doi.org/10.1109/TSE.2003.1245299Google ScholarDigital Library
Marcílio Mendonça, Thiago Tonelli Bartolomei, and Donald Cowan. 2008. Decision-making coordination in collaborative product configuration. In Proceedings of the ACM Symposium on Applied Computing (SAC’08). ACM, New York, NY, 108--113. DOI:https://doi.org/10.1145/1363686.1363715Google ScholarDigital Library
Breno Miranda, Emilio Cruciani, Roberto Verdecchia, and Antonia Bertolino. 2018. FAST approaches to scalable similarity-based test case prioritization. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). 222--232. DOI:https://doi.org/10.1145/3180155.3180210Google ScholarDigital Library
Siavash Mirarab, Soroush Akhlaghi, and Ladan Tahvildari. 2012. Size-constrained regression test case selection using multicriteria optimization. IEEE Trans. Softw. Eng. 38, 4 (2012), 936--956. DOI:https://doi.org/10.1109/TSE.2011.56Google ScholarDigital Library
Wiem Mkaouer, Marouane Kessentini, Adnan Shaout, Patrice Koligheu, Slim Bechikh, Kalyanmoy Deb, and Ali Ouni. 2015. Many-objective software remodularization using NSGA-III. ACM Trans. Softw. Eng. Methodol. 24, 3 (2015), 17:1–17:45. DOI:https://doi.org/10.1145/2729974Google ScholarDigital Library
Lev Nachmanson, Margus Veanes, Wolfram Schulte, Nikolai Tillmann, and Wolfgang Grieskamp. 2004. Optimal strategies for testing nondeterministic systems. In Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’04). 55--64. DOI:https://doi.org/10.1145/1007512.1007520Google ScholarDigital Library
George L. Nemhauser and Laurence A. Wolsey. 1988. Integer and Combinatorial Optimization. Wiley-Interscience, New York, NY.Google ScholarDigital Library
Jefferson Offutt, Jie Pan, and Jeffrey M. Voas. 1995. Procedures for reducing the size of coverage-based test sets. In Proceedings of the 12th International Conference on Testing Computer Software. 111--123.Google Scholar
Melih Özlen and Meral Azizoglu. 2009. Multi-objective integer programming: A general approach for generating all non-dominated solutions. Eur. J. Oper. Res. 199, 1 (2009), 25--35. DOI:https://doi.org/10.1016/j.ejor.2008.10.023Google ScholarCross Ref
Lili Pan, Junyi Li, Beiji Zou, and Hao Chen. 2005. Bi-objective model for test-suite reduction based on modified condition/decision coverage. In Proceedings of the 11th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC’05). 235--244. DOI:https://doi.org/10.1109/PRDC.2005.22Google ScholarDigital Library
Subhrakanta Panda and Durga Prasad Mohapatra. 2017. Regression test suite minimization using integer linear programming model. Softw. Pract. Exper. 47, 11 (2017), 1539--1560. DOI:https://doi.org/10.1002/spe.2485Google ScholarCross Ref
Annibale Panichella, Rocco Oliveto, Massimiliano Di Penta, and Andrea De Lucia. 2015. Improving multi-objective test case selection by injecting diversity in genetic algorithms. IEEE Trans. Softw. Eng. 41, 4 (2015), 358--383. DOI:https://doi.org/10.1109/TSE.2014.2364175Google ScholarDigital Library
Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE’15), Volume 1. 936--946. DOI:https://doi.org/10.1109/ICSE.2015.103Google ScholarCross Ref
Saeed Parsa and Alireza Khalilian. 2010. On the optimization approach towards test suite minimization. Int. J. Softw. Eng. Appl. 4, 1 (2010), 15--28.Google Scholar
Saeed Parsa, Alireza Khalilian, and Y. Fazlalizadeh. 2009. A new algorithm to test suite reduction based on cluster analysis. In Proceedings of the 2nd IEEE International Conference on Computer Science and Information Technology. 189--193. DOI:https://doi.org/10.1109/ICCSIT.2009.5234742Google Scholar
Tanay Kanti Paul and Man Fai Lau. 2014. A systematic literature review on modified condition and decision coverage. In Proceedings of the Symposium on Applied Computing (SAC’14). 1301--1308. DOI:https://doi.org/10.1145/2554850.2555004Google ScholarDigital Library
Adithya Abraham Philip, Ranjita Bhagwan, Rahul Kumar, Chandra Shekhar Maddila, and Nachiappan Nagappan. 2019. FastLane: Test minimization for rapidly deployed large-scale online services. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). 408--418. DOI:https://doi.org/10.1109/ICSE.2019.00054Google ScholarDigital Library
Nery Riquelme, Christian von Lücken, and Benjamín Barán. 2015. Performance metrics in multi-objective optimization. In Proceedings of the Latin American Computing Conference/Conferencia Latinoamericana En Informatica (CLEI’15). IEEE, 1--11.Google ScholarCross Ref
Gregg Rothermel and Mary Jean Harrold. 1993. A safe, efficient algorithm for regression test selection. In Proceedings of the Conference on Software Maintenance (ICSM’93). 358--367. DOI:https://doi.org/10.1109/ICSM.1993.366926Google ScholarCross Ref
Takfarinas Saber, Florian Delavernhe, Mike Papadakis, Michael O’Neill, and Anthony Ventresque. 2018. A hybrid algorithm for multi-objective test case selection. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC’18). 1--8. DOI:https://doi.org/10.1109/CEC.2018.8477875Google ScholarCross Ref
Sreedevi Sampath, Renée C. Bryce, and Atif M. Memon. 2013. A uniform representation of hybrid criteria for regression testing. IEEE Trans. Softw. Eng. 39, 10 (2013), 1326--1344. DOI:https://doi.org/10.1109/TSE.2013.16Google ScholarDigital Library
Sreedevi Sampath, Sara Sprenkle, Emily Gibson, Lori L. Pollock, and Amie L. Souter. 2005. Analyzing clusters of web application user sessions. ACM SIGSOFT Softw. Eng. Notes 30, 4 (2005), 1--7. DOI:https://doi.org/10.1145/1082983.1083255Google ScholarDigital Library
Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. 2013. On the value of user preferences in search-based software engineering: A case study in software product lines. In Proceedings of the 35th International Conference on Software Engineering (ICSE’13). 492--501. DOI:https://doi.org/10.1109/ICSE.2013.6606595Google ScholarDigital Library
Pierre-Yves Schobbens, Patrick Heymans, and Jean-Christophe Trigaux. 2006. Feature diagrams: A survey and a formal semantics. In Proceedings of the 14th IEEE International Conference on Requirements Engineering (RE’06). 136--145. DOI:https://doi.org/10.1109/RE.2006.23Google ScholarDigital Library
August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, and Darko Marinov. 2014. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’14). 246--256. DOI:https://doi.org/10.1145/2635868.2635921Google ScholarDigital Library
August Shi, Tifany Yung, Alex Gyori, and Darko Marinov. 2015. Comparing and combining test-suite reduction and regression test selection. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/SIGSOFT FSE’15). ACM, 237--247.Google ScholarDigital Library
Adam M. Smith and Gregory M. Kapfhammer. 2009. An empirical study of incorporating cost into test suite reduction and prioritization. In Proceedings of the ACM Symposium on Applied Computing (SAC’09). 461--467. DOI:https://doi.org/10.1145/1529282.1529382Google Scholar
Constantinos Stylianou and Andreas S. Andreou. 2014. Human Resource Allocation and Scheduling for Software Project Management. Springer Berlin, 73--106. DOI:https://doi.org/10.1007/978-3-642-55035-5_4Google Scholar
Sriraman Tallam and Neelam Gupta. 2005. A concept analysis inspired greedy algorithm for test suite minimization. In Proceedings of the ACM Workshop on Program Analysis For Software Tools and Engineering (PASTE’05). 35--42. DOI:https://doi.org/10.1145/1108792.1108802Google ScholarDigital Library
Marjan van den Akker, Sjaak Brinkkemper, Guido Diepen, and Johan Versendaal. 2008. Software product release planning through optimization and what-if analysis. Inf. Softw. Technol. 50, 1–2 (2008), 101--111. DOI:https://doi.org/10.1016/j.infsof.2007.10.017Google ScholarDigital Library
Pim van den Broek. 2010. Optimization of product instantiation using integer programming. In Proceedings of the Software Product Lines Conference Workshops (SPLC’10). Lancaster University, 107--112.Google Scholar
Vandana and Ajmer Singh. 2017. Multi-objective test case minimization using evolutionary algorithms: A review. In Proceedings of the International Conference of Electronics, Communication and Aerospace Technology (ICECA’17), Vol. 1. 329--334. DOI:https://doi.org/10.1109/ICECA.2017.8203698Google Scholar
Nadarajen Veerapen, Gabriela Ochoa, Mark Harman, and Edmund K. Burke. 2015. An integer linear programming approach to the single and bi-objective next release problem. Inf. Softw. Technol. 65 (2015), 1--13. DOI:https://doi.org/10.1016/j.infsof.2015.03.008Google ScholarDigital Library
Sergiy A. Vilkomir and Jonathan P. Bowen. 2001. Formalization of software testing criteria using the Z notation. In Proceedings of the 25th International Computer Software and Applications Conference (COMPSAC’01). 351--356. DOI:https://doi.org/10.1109/CMPSAC.2001.960638Google Scholar
Shuai Wang, Shaukat Ali, Tao Yue, Yan Li, and Marius Liaaen. 2016. A practical guide to select quality indicators for assessing Pareto-based search algorithms in search-based software engineering. In Proceedings of the 38th International Conference on Software Engineering (ICSE’16). 631--642. DOI:https://doi.org/10.1145/2884781.2884880Google ScholarDigital Library
Laurence A. Wolsey. 1998. Integer Programming. Wiley-Interscience, New York, NY.Google Scholar
Eric Wong, Joseph R. Horgan, Saul London, and Hiralal Agrawal. 1997. A study of effective regression testing in practice. In Proceedings of the 8th International Symposium on Software Reliability Engineering (ISSRE’97). 264--274. DOI:https://doi.org/10.1109/ISSRE.1997.630875Google ScholarCross Ref
Eric Wong, Joseph Robert Horgan, Aditya P. Mathur, and Alberto Pasquini. 1999. Test set size minimization and fault detection effectiveness: A case study in a space application. J. Syst. Softw. 48, 2 (1999), 79--89. DOI:https://doi.org/10.1016/S0164-1212(99)00048-5Google ScholarDigital Library
Yinxing Xue and Yan-Fu Li. 2018. Multi-objective integer programming approaches for solving optimal feature selection problem: A new perspective on multi-objective optimization problems in SBSE. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). 1231--1242. DOI:https://doi.org/10.1145/3180155.3180257Google ScholarDigital Library
Yinxing Xue 2018. IP-Method for Multi-objective Optimization in SPL. Retrieved from https://sites.google.com/view/ip-method-repsol/.Google Scholar
Shin Yoo and Mark Harman. 2010. Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation. J. Syst. Softw. 83, 4 (2010), 689--701. DOI:https://doi.org/10.1016/j.jss.2009.11.706Google ScholarDigital Library
Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: A survey. Softw. Test. Verif. Reliab. 22, 2 (Mar. 2012), 67--120. DOI:https://doi.org/10.1002/stv.430Google ScholarDigital Library
Shin Yoo, Mark Harman, and Shmuel Ur. 2011. Highly scalable multi-objective test suite minimisation using graphics cards. In Proceedings of the 3rd International Symposium on Search-based Software Engineering (SSBSE’11). 219--236. DOI:https://doi.org/10.1007/978-3-642-23716-4_20Google ScholarCross Ref
Chaoqiang Zhang, Alex Groce, and Mohammad Amin Alipour. 2014. Using test case reduction and prioritization to improve symbolic execution. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’14). 160--170. DOI:https://doi.org/10.1145/2610384.2610392Google ScholarDigital Library
Lu Zhang, Shan-Shan Hou, Chao Guo, Tao Xie, and Hong Mei. 2009. Time-aware test-case prioritization using integer linear programming. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’09). ACM, 213--224.Google ScholarDigital Library
Qingfu Zhang and Hui Li. 2007. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Trans. Evolut. Comput. 11, 6 (2007), 712--731.Google ScholarDigital Library
Yuanyuan Zhang, Mark Harman, and S. Afshin Mansouri. 2007. The multi-objective next release problem. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’07). 1129--1137. DOI:https://doi.org/10.1145/1276958.1277179Google Scholar
Yuanyuan Zhang, Mark Harman, Gabriela Ochoa, Guenther Ruhe, and Sjaak Brinkkemper. 2018. An empirical study of meta- and hyper-heuristic search for multi-objective release planning. ACM Trans. Softw. Eng. Methodol. 27, 1 (2018), 3:1–3:32. DOI:https://doi.org/10.1145/3196831Google ScholarDigital Library
Yikun Zhang, Jiceng Liu, Yingan Cui, Xinhong Hei, and Minghui Zhang. 2011. An improved quantum genetic algorithm for test suite reduction. In Proceedings of the IEEE International Conference on Computer Science and Automation Engineering, Vol. 2. 149--153. DOI:https://doi.org/10.1109/CSAE.2011.5952443Google ScholarCross Ref
Wei Zheng, Robert M. Hierons, Miqing Li, Xiaohui Liu, and Veronica Vinciotti. 2016. Multi-objective optimisation for regression testing. Inf. Sci. 334–335 (2016), 1--16. DOI:https://doi.org/10.1016/j.ins.2015.11.027Google Scholar
Wei Zheng, Xiaoxue Wu, Shichao Cao, and Jun Lin. 2018. MS-guided many-objective evolutionary optimisation for test suite minimisation. IET Softw. 12, 6 (2018), 547--554. DOI:https://doi.org/10.1049/iet-sen.2018.5133Google ScholarCross Ref
Eckart Zitzler and Lothar Thiele. 1999. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Trans. Evolut. Comput. 3, 4 (1999), 257--271. DOI:https://doi.org/10.1109/4235.797969Google ScholarDigital Library

Index Terms

Multi-objective Integer Programming Approaches for Solving the Multi-criteria Test-suite Minimization Problem: Towards Sound and Complete Solutions of a Particular Search-based Software-engineering Problem
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
  2. Software notations and tools
    1. Software maintenance tools

Recommendations

Multi-objective integer programming approaches for solving optimal feature selection problem: a new perspective on multi-objective optimization problems in SBSE
ICSE '18: Proceedings of the 40th International Conference on Software Engineering

The optimal feature selection problem in software product line is typically addressed by the approaches based on Indicator-based Evolutionary Algorithm (IBEA). In this study we first expose the mathematical nature of this problem --- multi-objective ...
Read More
Multi-objective integer programming approaches to Next Release Problem — Enhancing exact methods for finding whole pareto front
Abstract Context:
Project planning is a crucial part of software engineering, it involves selecting requirements to develop for the next release. How to make a good release plan is an optimization problem to maximize the goal of ...
Read More
Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation

Test suite minimisation techniques seek to reduce the effort required for regression testing by selecting a subset of test suites. In previous work, the problem has been considered as a single-objective optimisation problem. However, real world ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Transactions on Software Engineering and Methodology Volume 29, Issue 3
July 2020
292 pages
ISSN:1049-331X
EISSN:1557-7392
DOI:10.1145/3403667
Editor:
Mauro Pezzè
Università della Svizzera italiana and Università di Milano-Bicocca, Switzerland
Issue’s Table of Contents
Copyright © 2020 ACM
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 1 June 2020
- Online AM: 7 May 2020
- Revised: 1 March 2020
- Accepted: 1 March 2020
- Received: 1 May 2019
Published in tosem Volume 29, Issue 3

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
ε-constraint method
CWMOIP
Regression testing
big-M method
multi-objective integer programming
search-based software engineering
test-suite minimization
Qualifiers
- research-article
- Research
- Refereed
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 6
  Total Citations
  View Citations
- 3,859
  Total Downloads
- Downloads (Last 12 months)44
- Downloads (Last 6 weeks)6
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format .

View HTML Format

Multi-objective Integer Programming Approaches for Solving the Multi-criteria Test-suite Minimization Problem: Towards Sound and Complete Solutions of a Particular Search-based Software-engineering Problem

ACM Transactions on Software Engineering and Methodology

Abstract

References

Cited By

Index Terms

Recommendations

Multi-objective integer programming approaches for solving optimal feature selection problem: a new perspective on multi-objective optimization problems in SBSE

Multi-objective integer programming approaches to Next Release Problem — Enhancing exact methods for finding whole pareto front

Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation