Design and analysis of GUI test-case prioritization using weight-based methods

https://doi.org/10.1016/j.jss.2009.11.703Get rights and content

Abstract

Testing the correctness of a GUI-based application is more complex than the conventional code-based application. In addition to testing the underlying codes of the GUI application, the space of possible combinations of events with a large GUI-input sequence also requires creating numerous test cases to confirm the adequacy of the GUI testing. Running all GUI test cases and then fixing all found bugs may be time-consuming and delaying the project completion. Hence, it is important to advance the test cases that uncover the most faults as fast as possible in the testing process. Test-case prioritization has been proposed and used in recent years because it can improve the rate of fault detection during the testing phase. However, few studies have discussed the problem of GUI test-case prioritization. In this paper, we propose a weighted-event flow graph for solving the non-weighted GUI test case and ranking GUI test cases based on weight scores. The weighted scores can either be ranked from high to low or be ordered by dynamic adjusted scores. Finally, three experiments are performed, and experimental results show that the adjusted-weight method can obtain a better fault-detection rate.

Introduction

In recent years, more and more software systems have been built on event-driven software platforms. Event-driven software has a graphical user interface (GUI) which enables the user to either use mouse clicks or to key in data as input to change the event state (Jorgensen, 2007, Memon, 2001). Although the GUI facilitates the use of software, testing the correctness of a GUI-based application is more complex than testing conventional code-based applications. In addition to testing the underlying codes of the GUI application, the space of possible combinations of events with a large GUI-input sequence also requires creating numerous test cases to confirm the adequacy of the GUI testing (Memon et al., 2001a, Memon et al., 2001b).

Previous GUI testing research has discussed how to solve the problem of the exponential growth in test cases created by AI planning (Memon et al., 1999, Memon et al., 2001a, Memon et al., 2001b), Finite-State testing (White et al., 2001, Belli, 2001, Chen and Subramaniam, 2001), or Capture/Replay testing (Steven et al., 2000). Running all test cases and then fixing all found bugs may be time-consuming and delaying the project completion (Rothermel et al., 1999, Spillner et al., 2007, Dustin, 2003). Although there have been many non-GUI testing studies adopting a test-case prioritization criterion which can uncover more faults in the test period to facilitate more rapid repair of applications (Rothermel et al., 1999, Rothermel et al., 2001, Elbaum et al., 2002), few test-case prioritization methods considering GUI testing have been reported (Bryce and Memon, 2007, Belli et al., 2007).

In this paper, we will extend the non-weighted-event flow graph (EFG) (Memon et al., 2001a, Memon et al., 2001b) to a weighted-event flow graph (WEFG) for solving the non-weighted problem, and propose a weight-based method to prioritize the GUI test cases (GTC). The general prioritization techniques treat the importance of all event actions as identical, and depend on the test case length to prioritize ordering. According to this rule, the test case events of greater length are executed first. However, if a shorter test case event has more critical events than a longer one does, it may uncover more faults during testing, and will be assigned a higher priority for execution. If two test cases have the same length, the most common methods usually choose one of two test cases randomly. Both special conditions may have an impact on the rate of fault detection.

Given this, we propose three different weight-based methods for ordering the GTC priority, including the equal weight, fault-prone weight, and random weight. After calculating the weight of the GTC, we rank the weight-based methods by high-to-low and adjust these ways according to the weight scores. If the weight is large, many faults may be found by running this test case. Subsequently we analyze our proposed methods, and then compare them with other GTC prioritization approaches. For example, suppose there are two test cases, t1 and t2. Test case t1 includes events E1, E2, and E3, while t2 invokes E4 and E5. Based on our approach, each event will be assigned one weight in the beginning. The more important the event is, the higher the weight value (WV) is. Then, the weights of triggered events are summed up to be the execution priority of test cases. That is, the test case which the summation of all event weights is higher than the other test cases can detect the most faults in the beginning of test-case execution. If the weights of all events are the same, then the summation of weights for t1 is higher than that of t2. This represents the supposition that t1 could detect more quickly the most faults. However, based on the WV for each event type, we assume E1 = 1, E2 = 2, E3 = 3, E4 = 5, E5 = 5. The weight of t1 is 6 and t2 is 10, so the rate of fault detection for t2 is higher than that of t1.

Finally, we also developed a prioritization tool called ATCPAT (Automated Test-Case Prioritization Analysis Tool) to rapidly obtain the fault-detection rate using many prioritization techniques. Our experimental results show that the fault-prone weight with adjusted-weight ordering provides a better rate of fault detection in GUI applications.

The rest of this paper is organized follows. In Section 2, we introduce test-case prioritization techniques based on GUI testing, and review related works. Section 3 includes the proposed weight-based GTC prioritization technique and explains how to create WEFGs and to assign event weights in GTC. Section 4 demonstrates the test-case prioritization tool and shows the experimental results. Finally, the conclusions will be given in Section 5.

Section snippets

GUI testing and test-case prioritization

In the software testing process, testers have to ensure the correctness of software by using some testing techniques. Before starting a testing procedure, we should determine the criteria for knowing the adequacy of testing software (Zhu et al., 1997). The criteria include code coverage such as statement, branch coverage, or MC/DC for white-box testing, or specification or requirements satisfaction for black-box testing. When the criterion has been decided, the next step is to create the test

Creating a weighted-event flow graph

In order to produce a WEFG, we extend the construction of an EFG. Before assigning a weight to each event, all events are classified into one of the following five events (Memon and Xie, 2004a, Memon and Xie, 2004b): restricted-focus events, unrestricted-focus events, termination events, menu-open events, and system-interaction events. These classifications can be used to identify components in the GUI application. The weight of each event type will be decided next.

If the importance of each

Criteria for comparisons

In this paper, the Average Percentage Faults Detected (APFD) was used to measure the percentage of faults detected over the testing process. APFD is defined as (Rothermel et al., 2001):APFD=1-TF1+TF2++TFmnm+12,where n is the number of test cases in the test suite T, m is the total number of faults in the software under test, F is a subset of m faults detected by T, and TFi is the first test case in the new ordering T′ that reveals the fault i. The higher APFD value means that the ranked test

Conclusions and future works

The problems of test-case prioritization have been widely investigated to improve the effectiveness of fault detection, but few studies have focused on GTC prioritization. General GTC prioritization techniques such as the unique event coverage approach and the untreated method sometimes impact the performance of fault detection when prioritizing test cases using random tie-breaking, so that test cases with more important events may be executed later. In this paper, to fix this problem, we

Acknowledgments

The work described in this paper was supported by the National Science Council, Taiwan, under Grant NSC 97-2221-E-007-052-MY3 and NSC 98-2221-E-007-067, and also substantially supported by a grant from the Ministry of Economic Affairs of Taiwan (Project No. 98-EC-17-A-02-S2-0097). The authors are pleased to thank Professor Atif M. Memon of University of Maryland for his many constructive and insightful suggestions for improving the details of this paper. Besides, we also thank several anonymous

Chin-Yu Huang is currently an Associate Professor in the Department of Computer Science at National Tsing Hua University, Hsinchu, Taiwan. He received the MS (1994), and the Ph.D. (2000) in Electrical Engineering from National Taiwan University, Taipei. He was with the Bank of Taiwan from 1994 to 1999, and was a senior software engineer at Taiwan Semiconductor Manufacturing Company from 1999 to 2000. Before joining NTHU in 2003, he was a division chief of the Central Bank of China, Taipei. He

References (44)

  • D. Jeffrey et al.

    Experiments with test case prioritization using relevant slices

    Journal of Systems and Software

    (2008)
  • M. Jorgensen et al.

    Software effort estimation by analogy and “regression toward the mean”

    Journal of Systems and Software

    (2003)
  • J.W. Lin et al.

    Analysis of test suite reduction with enhanced tie-breaking techniques

    Information and Software Technology

    (2009)
  • P. Ammann et al.

    Introduction to Software Testing

    (2008)
  • Belli, F., 2001. Finite-state testing and analysis of graphical user interfaces. In: Proceedings of 12th International...
  • Belli, F., Eminov, M., Gokce, N., 2007. Prioritizing coverage-oriented testing process – an adaptive-learning-based...
  • Bryce, R.C., Memon, A.M., 2007. Test suite prioritization by interaction coverage. In: Proceedings of the Workshop on...
  • Chen, J., Subramaniam, S., 2001. A GUI environment to manipulate FSMs for testing GUI-based applications in Java. In:...
  • E. Dustin

    Effective Software Testing: 50 Specific Ways to Improve Your Testing

    (2003)
  • S. Elbaum et al.

    Test case prioritization: a family of empirical studies

    IEEE Transactions on Software Engineering

    (2002)
  • M. Grindal et al.

    Combination testing strategies: a survey

    Software Testing, Verification and Reliability

    (2005)
  • GUITAR, 2009. This tool can be obtained from <http://www.cs.umd.edu/~atif/GUITAR-distribution/manuals/index.html> and...
  • Hackner, D., Memon, A.M., 2008. Test case generator for GUITAR. In: Proceedings of the 30th International Conference on...
  • M.J. Harrold et al.

    A methodology for controlling the size of a test suite

    ACM Transactions on Software Engineering and Methodology (TOSEM)

    (1993)
  • C.Y. Huang et al.

    Optimal release time for software systems considering cost, testing-effort, and test efficiency

    IEEE Transactions on Reliability

    (2005)
  • C.Y. Huang et al.

    Optimal testing resource allocation and sensitivity analysis in software development

    IEEE Transactions on Reliability

    (2005)
  • Java Swing Calculator, 2009. This application can be obtained from...
  • P.C. Jorgensen

    Software Testing: A Craftsman’s Approach

    (2007)
  • C. Kaner et al.

    Testing Computer Software

    (1999)
  • Ma, Y.S., Offutt, J., Kwon, Y.R., 2006. MuJava: A mutation system for Java. In: Proceedings of the 28th International...
  • Memon, A.M., 2001. A comprehensive framework for testing graphical user interfaces. Ph.D. Thesis, Department of...
  • Memon, A.M., Xie, Q., 2004a. Empirical evaluation of the fault-detection effectiveness of smoke regression test cases...
  • Cited by (35)

    • Evaluation and analysis of incorporating Fuzzy Expert System approach into test suite reduction

      2016, Information and Software Technology
      Citation Excerpt :

      But it is worth noting that the fault detection capability for the Fuzzy-GRE algorithm is increased by 7.21%. Finally, it should be noted that GUI testing is inherently more difficult than traditional code-based program testing [81,82]. Testing the correctness of a GUI-based application is more complex than the conventional code-based application.

    • Test Case Prioritization for Mobile Apps

      2022, Proceedings - 2022 International Conference on Computational Science and Computational Intelligence, CSCI 2022
    • Test Case Prioritization Approach for Sequence of Events Using Complexity Factor

      2021, International Journal on Advanced Science, Engineering and Information Technology
    • Model-based Testing of Interactive Systems using Interaction Sequences

      2020, Proceedings of the ACM on Human-Computer Interaction
    View all citing articles on Scopus

    Chin-Yu Huang is currently an Associate Professor in the Department of Computer Science at National Tsing Hua University, Hsinchu, Taiwan. He received the MS (1994), and the Ph.D. (2000) in Electrical Engineering from National Taiwan University, Taipei. He was with the Bank of Taiwan from 1994 to 1999, and was a senior software engineer at Taiwan Semiconductor Manufacturing Company from 1999 to 2000. Before joining NTHU in 2003, he was a division chief of the Central Bank of China, Taipei. He received the Ta-You Wu Memorial Award from the National Science Council of Taiwan in 2008. His research interests are software reliability engineering, software testing, software metrics, software testability, fault tree analysis, and system safety assessment.

    Jun-Ru Chang received the B.S. (2004) in Computer Science and Information Engineering from National Chiao Tung University and the MS (2006) from the Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan. He is currently pursuing his Ph.D. degree in the Institute of Information Systems and Applications at National Tsing Hua University. His research interests include software testing, software reliability and embedded system.

    Yung-Hsin Chang received the B.S. (2006) in Computer Science and Engineering from Yuan Ze University and received the MS from the Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu, Taiwan, in 2008. Currently she is working at ASUSTeK Computer Inc., Taipei, Taiwan.

    View full text