Design and analysis of GUI test-case prioritization using weight-based methods
Introduction
In recent years, more and more software systems have been built on event-driven software platforms. Event-driven software has a graphical user interface (GUI) which enables the user to either use mouse clicks or to key in data as input to change the event state (Jorgensen, 2007, Memon, 2001). Although the GUI facilitates the use of software, testing the correctness of a GUI-based application is more complex than testing conventional code-based applications. In addition to testing the underlying codes of the GUI application, the space of possible combinations of events with a large GUI-input sequence also requires creating numerous test cases to confirm the adequacy of the GUI testing (Memon et al., 2001a, Memon et al., 2001b).
Previous GUI testing research has discussed how to solve the problem of the exponential growth in test cases created by AI planning (Memon et al., 1999, Memon et al., 2001a, Memon et al., 2001b), Finite-State testing (White et al., 2001, Belli, 2001, Chen and Subramaniam, 2001), or Capture/Replay testing (Steven et al., 2000). Running all test cases and then fixing all found bugs may be time-consuming and delaying the project completion (Rothermel et al., 1999, Spillner et al., 2007, Dustin, 2003). Although there have been many non-GUI testing studies adopting a test-case prioritization criterion which can uncover more faults in the test period to facilitate more rapid repair of applications (Rothermel et al., 1999, Rothermel et al., 2001, Elbaum et al., 2002), few test-case prioritization methods considering GUI testing have been reported (Bryce and Memon, 2007, Belli et al., 2007).
In this paper, we will extend the non-weighted-event flow graph (EFG) (Memon et al., 2001a, Memon et al., 2001b) to a weighted-event flow graph (WEFG) for solving the non-weighted problem, and propose a weight-based method to prioritize the GUI test cases (GTC). The general prioritization techniques treat the importance of all event actions as identical, and depend on the test case length to prioritize ordering. According to this rule, the test case events of greater length are executed first. However, if a shorter test case event has more critical events than a longer one does, it may uncover more faults during testing, and will be assigned a higher priority for execution. If two test cases have the same length, the most common methods usually choose one of two test cases randomly. Both special conditions may have an impact on the rate of fault detection.
Given this, we propose three different weight-based methods for ordering the GTC priority, including the equal weight, fault-prone weight, and random weight. After calculating the weight of the GTC, we rank the weight-based methods by high-to-low and adjust these ways according to the weight scores. If the weight is large, many faults may be found by running this test case. Subsequently we analyze our proposed methods, and then compare them with other GTC prioritization approaches. For example, suppose there are two test cases, t1 and t2. Test case t1 includes events E1, E2, and E3, while t2 invokes E4 and E5. Based on our approach, each event will be assigned one weight in the beginning. The more important the event is, the higher the weight value (WV) is. Then, the weights of triggered events are summed up to be the execution priority of test cases. That is, the test case which the summation of all event weights is higher than the other test cases can detect the most faults in the beginning of test-case execution. If the weights of all events are the same, then the summation of weights for t1 is higher than that of t2. This represents the supposition that t1 could detect more quickly the most faults. However, based on the WV for each event type, we assume E1 = 1, E2 = 2, E3 = 3, E4 = 5, E5 = 5. The weight of t1 is 6 and t2 is 10, so the rate of fault detection for t2 is higher than that of t1.
Finally, we also developed a prioritization tool called ATCPAT (Automated Test-Case Prioritization Analysis Tool) to rapidly obtain the fault-detection rate using many prioritization techniques. Our experimental results show that the fault-prone weight with adjusted-weight ordering provides a better rate of fault detection in GUI applications.
The rest of this paper is organized follows. In Section 2, we introduce test-case prioritization techniques based on GUI testing, and review related works. Section 3 includes the proposed weight-based GTC prioritization technique and explains how to create WEFGs and to assign event weights in GTC. Section 4 demonstrates the test-case prioritization tool and shows the experimental results. Finally, the conclusions will be given in Section 5.
Section snippets
GUI testing and test-case prioritization
In the software testing process, testers have to ensure the correctness of software by using some testing techniques. Before starting a testing procedure, we should determine the criteria for knowing the adequacy of testing software (Zhu et al., 1997). The criteria include code coverage such as statement, branch coverage, or MC/DC for white-box testing, or specification or requirements satisfaction for black-box testing. When the criterion has been decided, the next step is to create the test
Creating a weighted-event flow graph
In order to produce a WEFG, we extend the construction of an EFG. Before assigning a weight to each event, all events are classified into one of the following five events (Memon and Xie, 2004a, Memon and Xie, 2004b): restricted-focus events, unrestricted-focus events, termination events, menu-open events, and system-interaction events. These classifications can be used to identify components in the GUI application. The weight of each event type will be decided next.
If the importance of each
Criteria for comparisons
In this paper, the Average Percentage Faults Detected (APFD) was used to measure the percentage of faults detected over the testing process. APFD is defined as (Rothermel et al., 2001):where n is the number of test cases in the test suite T, m is the total number of faults in the software under test, F is a subset of m faults detected by T, and TFi is the first test case in the new ordering T′ that reveals the fault i. The higher APFD value means that the ranked test
Conclusions and future works
The problems of test-case prioritization have been widely investigated to improve the effectiveness of fault detection, but few studies have focused on GTC prioritization. General GTC prioritization techniques such as the unique event coverage approach and the untreated method sometimes impact the performance of fault detection when prioritizing test cases using random tie-breaking, so that test cases with more important events may be executed later. In this paper, to fix this problem, we
Acknowledgments
The work described in this paper was supported by the National Science Council, Taiwan, under Grant NSC 97-2221-E-007-052-MY3 and NSC 98-2221-E-007-067, and also substantially supported by a grant from the Ministry of Economic Affairs of Taiwan (Project No. 98-EC-17-A-02-S2-0097). The authors are pleased to thank Professor Atif M. Memon of University of Maryland for his many constructive and insightful suggestions for improving the details of this paper. Besides, we also thank several anonymous
Chin-Yu Huang is currently an Associate Professor in the Department of Computer Science at National Tsing Hua University, Hsinchu, Taiwan. He received the MS (1994), and the Ph.D. (2000) in Electrical Engineering from National Taiwan University, Taipei. He was with the Bank of Taiwan from 1994 to 1999, and was a senior software engineer at Taiwan Semiconductor Manufacturing Company from 1999 to 2000. Before joining NTHU in 2003, he was a division chief of the Central Bank of China, Taipei. He
References (44)
- et al.
Experiments with test case prioritization using relevant slices
Journal of Systems and Software
(2008) - et al.
Software effort estimation by analogy and “regression toward the mean”
Journal of Systems and Software
(2003) - et al.
Analysis of test suite reduction with enhanced tie-breaking techniques
Information and Software Technology
(2009) - et al.
Introduction to Software Testing
(2008) - Belli, F., 2001. Finite-state testing and analysis of graphical user interfaces. In: Proceedings of 12th International...
- Belli, F., Eminov, M., Gokce, N., 2007. Prioritizing coverage-oriented testing process – an adaptive-learning-based...
- Bryce, R.C., Memon, A.M., 2007. Test suite prioritization by interaction coverage. In: Proceedings of the Workshop on...
- Chen, J., Subramaniam, S., 2001. A GUI environment to manipulate FSMs for testing GUI-based applications in Java. In:...
Effective Software Testing: 50 Specific Ways to Improve Your Testing
(2003)- et al.
Test case prioritization: a family of empirical studies
IEEE Transactions on Software Engineering
(2002)
Combination testing strategies: a survey
Software Testing, Verification and Reliability
A methodology for controlling the size of a test suite
ACM Transactions on Software Engineering and Methodology (TOSEM)
Optimal release time for software systems considering cost, testing-effort, and test efficiency
IEEE Transactions on Reliability
Optimal testing resource allocation and sensitivity analysis in software development
IEEE Transactions on Reliability
Software Testing: A Craftsman’s Approach
Testing Computer Software
Cited by (35)
A taxonomy of metrics for GUI-based testing research: A systematic literature review
2022, Information and Software TechnologyEvaluation and analysis of incorporating Fuzzy Expert System approach into test suite reduction
2016, Information and Software TechnologyCitation Excerpt :But it is worth noting that the fault detection capability for the Fuzzy-GRE algorithm is increased by 7.21%. Finally, it should be noted that GUI testing is inherently more difficult than traditional code-based program testing [81,82]. Testing the correctness of a GUI-based application is more complex than the conventional code-based application.
Design of a Novel Weighted-Multicriteria Analysis Model for Effective Test Case Prioritization for Network and Robotic Projects
2022, Wireless Personal CommunicationsTest Case Prioritization for Mobile Apps
2022, Proceedings - 2022 International Conference on Computational Science and Computational Intelligence, CSCI 2022Test Case Prioritization Approach for Sequence of Events Using Complexity Factor
2021, International Journal on Advanced Science, Engineering and Information TechnologyModel-based Testing of Interactive Systems using Interaction Sequences
2020, Proceedings of the ACM on Human-Computer Interaction
Chin-Yu Huang is currently an Associate Professor in the Department of Computer Science at National Tsing Hua University, Hsinchu, Taiwan. He received the MS (1994), and the Ph.D. (2000) in Electrical Engineering from National Taiwan University, Taipei. He was with the Bank of Taiwan from 1994 to 1999, and was a senior software engineer at Taiwan Semiconductor Manufacturing Company from 1999 to 2000. Before joining NTHU in 2003, he was a division chief of the Central Bank of China, Taipei. He received the Ta-You Wu Memorial Award from the National Science Council of Taiwan in 2008. His research interests are software reliability engineering, software testing, software metrics, software testability, fault tree analysis, and system safety assessment.
Jun-Ru Chang received the B.S. (2004) in Computer Science and Information Engineering from National Chiao Tung University and the MS (2006) from the Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan. He is currently pursuing his Ph.D. degree in the Institute of Information Systems and Applications at National Tsing Hua University. His research interests include software testing, software reliability and embedded system.
Yung-Hsin Chang received the B.S. (2006) in Computer Science and Engineering from Yuan Ze University and received the MS from the Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu, Taiwan, in 2008. Currently she is working at ASUSTeK Computer Inc., Taipei, Taiwan.