skip to main content
research-article

Automatically repairing event sequence-based GUI test suites for regression testing

Published: 12 November 2008 Publication History

Abstract

Although graphical user interfaces (GUIs) constitute a large part of the software being developed today and are typically created using rapid prototyping, there are no effective regression testing techniques for GUIs. The needs of GUI regression testing differ from those of traditional software. When the structure of a GUI is modified, test cases from the original GUI's suite are either reusable or unusable on the modified GUI. Because GUI test case generation is expensive, our goal is to make the unusable test cases usable, thereby helping to retain the suite's event coverage. The idea of reusing these unusable (obsolete) test cases has not been explored before. This article shows that a large number of test cases become unusable for GUIs. It presents a new GUI regression testing technique that first automatically determines the usable and unusable test cases from a test suite after a GUI modification, then determines the unusable test cases that can be repaired so that they can execute on the modified GUI, and finally uses repairing transformations to repair the test cases. This regression testing technique along with four repairing transformations has been implemented. An empirical study for four open-source applications demonstrates that (1) this approach is effective in that many of the test cases can be repaired, and is practical in terms of its time performance, (2) certain types of test cases are more prone to becoming unusable, and (3) certain types of “dominator” events, when modified, make a large number of test cases unusable.

References

[1]
Abbot. 2003. Abbot Java GUI Test Framework. Available online at http://abbot.sourceforge.net.
[2]
Agrawal, H., Horgan, J. R., Krauser, E. W., and London, S. A. 1993. Incremental regression testing. In Proceedings of the Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 348--357.
[3]
Aho, A. V., Sethi, R., and Ullman, J. D. 1986. Compilers Principles, Techniques, and Tools. Addison-Wesley, Reading, MA.
[4]
Beizer, B. 1990. Software Testing Techniques, 2nd ed. Van Nostrand Reinhold, New York, NY.
[5]
Benedusi, P., Cimitile, A., and DeCarlini, U. 1988. Post-maintenance testing based on path change analysis. In Proceedings of the IEEE Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 352--368.
[6]
Binkley, D. 1997. Semantics guided regression test cost reduction. IEEE Trans. Softw. Eng. 23, 8 (Aug.), 498--516.
[7]
Hammontree, M. L., Hendrickson, J. J., and Hensley, B. W. 1992. Integrated data capture and analysis tools for research and testing an graphical user interfaces. In Proceedings of the Conference on Human Factors in Computing Systems. ACM Press, New York, NY, 431--432.
[8]
Harrold, M. J., Gupta, R., and Soffa, M. L. 1993. A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Meth. 2, 3 (July), 270--285.
[9]
Harrold, M. J., McGregor, J. D., and Fitzpatrick, K. J. 1992. Incremental testing of object-oriented class structures. In Proceedings of the 14th International Conference on Software Engineering. ACM Press, New York, NY, 68--80.
[10]
Harrold, M. J. and Soffa, M. L. 1989. Interprocedual data flow testing. In Proceedings of the ACM SIGSOFT '89 Third Symposium on Testing, Analysis, and Verification (TAV3). ACM Press, New York, NY, 158--167.
[11]
Hicinbothom, J. H. and Zachary, W. W. 1993. A tool for automatically generating transcripts of human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. SPECIAL SESSIONS: Demonstrations, vol. 2. ACM Press, New York, NY, 1042.
[12]
JUnitResources. 2005. JUnit, testing resources for extreme programming. Available online at http://junit.org/news/extension/gui/index.htm.
[13]
Kasik, D. J. and George, H. G. 1996. Toward automatic generation of novice user test scripts. In Proceedings of the Conference on Human Factors in Computing Systems: Common Ground. ACM Press, New York, NY, 244--251.
[14]
Kepple, L. R. 1992. A new paradigm for cross-platform automated GUI testing. X Res. 3, 1 (June), 155--178.
[15]
Kepple, L. R. 1994. The black art of GUI testing. Dr. Dobb's J. Softw. Tools 19, 2 (Feb.), 40.
[16]
Kung, D. C., Gao, J., Hsia, P., Toyoshima, Y., and Chen, C. 1996. On regression testing of object-oriented programs. J. Syst. Softw. 32, 1 (Jan.), 21--31.
[17]
Lengauer, T. and Tarjan, R. E. 1979. A fast algorithm for finding dominators in a flowgraph. ACM Trans. Program. Lang. Syst. 1, 1, 121--141.
[18]
Marick, B. 2002. Bypassing the GUI. Softw. Test. Qual. Eng. Mag. 2, 41--47.
[19]
Memon, A., Banerjee, I., and Nagarajan, A. 2003. GUI ripping: Reverse engineering of graphical user interfaces for testing. In WCRE '03: Proceedings of the 10th Working Conference on Reverse Engineering. IEEE Computer Society Press, Los Alamitos, CA, 260--269.
[20]
Memon, A., Nagarajan, A., and Xie, Q. 2005. Automating regression testing for evolving GUI software. J. Softw. Maint. Evolut. Res. Pract. 17, 1, 27--64.
[21]
Memon, A. M. 2001. A comprehensive framework for testing graphical user interfaces. Ph.D. dissertation, Department of Computer Science, University of Pittsburgh, Pittsburgh, PA.
[22]
Memon, A. M. 2002. GUI testing: Pitfalls and process. IEEE Comput. 35, 8 (Aug.), 90--91.
[23]
Memon, A. M., Pollack, M. E., and Soffa, M. L. 2000. Automated test oracles for GUIs. In Proceedings of the ACM SIGSOFT 8th International Symposium on the Foundations of Software Engineering (FSE-8). ACM Press, New York, NY, 30--39.
[24]
Memon, A. M., Pollack, M. E., and Soffa, M. L. 2001a. Hierarchical GUI test case generation using automated planning. IEEE Trans. Softw. Eng. 27, 2 (Feb.), 144--155.
[25]
Memon, A. M. and Soffa, M. L. 2003. Regression testing of GUIs. In ESEC/FSE-11: Proceedings of the 9th European Software Engineering Conference Held Jointly with 11th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY, 118--127.
[26]
Memon, A. M., Soffa, M. L., and Pollack, M. E. 2001b. Coverage criteria for GUI testing. In Proceedings of the 8th European Software Engineering Conference (ESEC) and 9th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE-9). ACM Press, New York, NY, 256--267.
[27]
Memon, A. M. and Xie, Q. 2005. Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software. IEEE Trans. Softw. Eng. 31, 10 (Oct.), 884--896.
[28]
Myers, B. A. 1995. User interface software tools. ACM Trans. Comput.-Hum. Interact. 2, 1, 64--103.
[29]
Onoma, A. K., Tsai, W.-T., Poonawala, M., and Suganuma, H. 1998. Regression testing in an industrial environment. Commun. ACM 41, 5, 81--86.
[30]
Ostrand, T., Anodide, A., Foster, H., and Goradia, T. 1998. A visual test development environment for GUI systems. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA-98). ACM Press, New York, NY, 82--92.
[31]
Pacheco, C., Lahiri, S. K., Ernst, M. D., and Ball, T. 2007. Feedback-directed random test generation. In ICSE'07, Proceedings of the 29th International Conference on Software Engineering. ACM Press, New York, NY.
[32]
Pollock, L. and Soffa, M. L. 1992. Incremental global reoptimization of programs. ACM Trans. Programm. Lang. Syst. 14, 2 (Apr.), 173--200.
[33]
RationalRobot. 2003. Rational Robot. Available online at http://www.rational.com.ar/tools/robot.html.
[34]
Rosenberg, D. 1993. User interface prototyping paradigms in the 90's. In Proceedings of the ACM INTERCHI'93 Conference on Human Factors in Computing Systems—Adjunct Proceedings. Tutorials. ACM Press, New York, NY, 231.
[35]
Rosenblum, D. and Rothermel, G. 1997. A comparative study of regression test selection techniques. In Proceedings of the IEEE Computer Society 2nd International Workshop on Empirical Studies of Software maintenance. IEEE Computer Society Press, Los Alamitos, CA, 89--94.
[36]
Rosenblum, D. S. 1995. A practical approach to programming with assertions. IEEE Trans. Softw. Eng. 21, 1, 19--31.
[37]
Rosenblum, D. S. and Weyuker, E. J. 1997. Using coverage information to predict the cost-effectiveness of regression testing strategies. IEEE Trans. Softw. Eng. 23, 3 (Mar.), 146--156.
[38]
Rothermel, G. and Harrold, M. J. 1993. A safe, efficient algorithm for regression test selection. In Proceedings of the Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 358--369.
[39]
Rothermel, G. and Harrold, M. J. 1997. A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. and Method. 6, 2 (Apr.), 173--210.
[40]
Rothermel, G. and Harrold, M. J. 1998. Empirical studies of a safe regression test selection technique. IEEE Trans. Softw. Eng. 24, 6 (June), 401--419.
[41]
Rothermel, G., Harrold, M. J., Ostrin, J., and Hong, C. 1998. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings; International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 34--43.
[42]
Shehady, R. K. and Siewiorek, D. P. 1997. A method to automate user interface testing using variable finite state machines. In Proceedings of the twenty-Seventh Annual International Symposium on Fault-Tolerant Computing (FTCS'97). IEEE Press, Los Alamitos, CA, 80--88.
[43]
Software Research, Inc., Capture-Replay Tool 2003. Software Research, Inc., Capture-Replay Tool. Available online at http://soft.com.
[44]
Thatcher, J. 1994. Screen reader/2--programmed access to the GUI. In ICCHP '94: Proceedings of the 4th International Conference on Computers for Handicapped Persons. Springer-Verlag New York, NY, 76--88.
[45]
Voas, J. 1997. How assertions can increase test effectiveness. IEEE Softw. 14, 2, 118--119,122.
[46]
Walworth, A. 1997. Java GUI testing. Dr. Dobb's J. Softw. Tools 22, 2 (Feb.), 30, 32, 34.
[47]
White, L. 1996. Regression testing of GUI event interactions. In Proceedings of the International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 350--358.
[48]
White, L. and Almezen, H. 2000. Generating test cases for GUI responsibilities using complete interaction sequences. In Proceedings of the International Symposium on Software Reliability Engineering. IEEE Computer Society Press, Los Alamitos, CA, 110--121.
[49]
White, L., Almezen, H., and Sastry, S. 2003. Firewall regression testing of gui sequences and their interactions. In ICSM '03: Proceedings of the International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 398.
[50]
White, L., Jaber, K., and Robinson, B. 2005. Utilization of extended firewall for object-oriented regression testing. In ICSM '05: Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05). IEEE Computer Society Press, Los Alamitos, CA, 695--698.
[51]
WinRunner. 2003. Mercury Interactive WinRunner. Available online at http://www.mercuryinteractive.com/products/winrunner.
[52]
Wittel, Jr., W. I. and Lewis, T. G. 1991. Integrating the MVC paradigm into an object-oriented framework to accelerate GUI application development. Tech. rep. 91-60-06. Department of Computer Science, Oregon State University, Carvallis, OR.
[53]
Wohlin, C., Runeson, P., Host, M., Ohlsson, M. C., Regnell, B., and Wesslen, A. 2000. Experimentation in software engineering: An introduction. Kluwer Academic Publishers, Norwell, MA.
[54]
Xie, Q. and Memon, A. M. 2006. Model-based testing of community-driven open-source GUI applications. In Proceedings of the 22nd IEEE International Conference on Software Maintenance. IEEE Computer Society, Los Alamitos, CA, 145--154.
[55]
Xie, Q. and Memon, A. M. 2007. Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Test. Method. 16, 1, 4.
[56]
Yuan, X. and Memon, A. M. 2007. Using GUI run-time state as feedback to generate test cases. In ICSE'07, Proceedings of the 29th International Conference on Software Engineering. ACM Press, New York, NY.

Cited By

View all
  • (2024)Repairing Obsolete GUI Test Scripts for Android Applications with Exploration and BacktrackingProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671377(51-60)Online publication date: 24-Jul-2024
  • (2024)Comprehensive Semantic Repair of Obsolete GUI Test Scripts for Mobile ApplicationsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639108(1-13)Online publication date: 20-May-2024
  • (2024)Investigating the robustness of locators in template-based Web application testing using a GUI change classification modelJournal of Systems and Software10.1016/j.jss.2023.111932210:COnline publication date: 1-Apr-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology
ACM Transactions on Software Engineering and Methodology  Volume 18, Issue 2
November 2008
130 pages
ISSN:1049-331X
EISSN:1557-7392
DOI:10.1145/1416563
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2008
Accepted: 01 September 2007
Revised: 01 March 2007
Received: 01 January 2006
Published in TOSEM Volume 18, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Graphical user interfaces
  2. regression testing
  3. repairing test cases
  4. test case management
  5. test maintenance

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)22
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Repairing Obsolete GUI Test Scripts for Android Applications with Exploration and BacktrackingProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671377(51-60)Online publication date: 24-Jul-2024
  • (2024)Comprehensive Semantic Repair of Obsolete GUI Test Scripts for Mobile ApplicationsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639108(1-13)Online publication date: 20-May-2024
  • (2024)Investigating the robustness of locators in template-based Web application testing using a GUI change classification modelJournal of Systems and Software10.1016/j.jss.2023.111932210:COnline publication date: 1-Apr-2024
  • (2023)Semantic Test Repair for Web ApplicationsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616324(1190-1202)Online publication date: 30-Nov-2023
  • (2023)DeltaDroid: Dynamic Delivery Testing in AndroidACM Transactions on Software Engineering and Methodology10.1145/356321332:4(1-26)Online publication date: 26-May-2023
  • (2023)AdapTV+: Enhancing Model-Based Test Adaptation for Smart TVs through Icon Recognition2023 IEEE 28th Pacific Rim International Symposium on Dependable Computing (PRDC)10.1109/PRDC59308.2023.00033(209-216)Online publication date: 24-Oct-2023
  • (2023)Automated Fixing of Web UI Tests via Iterative Element MatchingProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00048(1188-1199)Online publication date: 11-Nov-2023
  • (2023)AdapTV: A Model-Based Test Adaptation Approach for End-to-End User Interface Testing of Smart TVsIEEE Access10.1109/ACCESS.2023.326274611(32095-32118)Online publication date: 2023
  • (2022)Leveraging code-test co-evolution patterns for automated test case recommendationProceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test10.1145/3524481.3527222(65-76)Online publication date: 17-May-2022
  • (2022)Patterns of Code-to-Test Co-evolution for Automated Test Suite Maintenance2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00023(116-127)Online publication date: Apr-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media