Skip to main content
SpringerLink
Log in

GUITAR: an innovative tool for automated testing of GUI-driven software

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Most of today’s software applications feature a graphical user interface (GUI) front-end. System testing of these applications requires that test cases, modeled as sequences of GUI events, be generated and executed on the software. We term GUI testing as the process of testing a software application through its GUI. Researchers and practitioners agree that one must employ a variety of techniques (e.g., model-based, capture/replay, manually scripted) for effective GUI testing. Yet, the tools available today for GUI testing are limited in the techniques they support. In this paper, we describe an innovative tool called GUITAR that supports a wide variety of GUI testing techniques. The innovation lies in the architecture of GUITAR, which uses plug-ins to support flexibility and extensibility. Software developers and quality assurance engineers may use this architecture to create new toolchains, new workflows based on the toolchains, and plug in a variety of measurement tools to conduct GUI testing. We demonstrate these features of GUITAR via several carefully crafted case studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://www.jfcunit.sourceforge.net.

  2. http://www.seleniumhq.org/projects/webdriver.

  3. http://www.code.google.com/p/robotium.

  4. http://www.abbot.sourceforge.net.

  5. http://www.parasoft.com/jsp/products/soatest.jsp.

  6. http://www.testautomationfx.com.

  7. http://www.seleniumhq.org/projects/ide.

  8. http://www.hp.com/QuickTestPro.

  9. http://www.junit.org.

  10. http://www.ibm.com/software/awdtools/tester/robot.

  11. http://www.hp.com/functionaltesting.

  12. http://developer.android.com/guide/developing/tools/monkey.html.

  13. http://www.bredex.de/web/index.php/guidancer_jubula_en.html.

  14. http://nmodel.codeplex.com/.

  15. http://www.graphviz.org/.

  16. https://gephi.org.

  17. http://argouml.tigris.org.

  18. http://jabref.sourceforge.net.

  19. Standard GUI terminology; see detailed explanations at msdn.microsoft.com/library/en-us/vbcon/html/vbtskdisplayingmodelessform.asp and documents.wolfram.com/v4/AddOns/JLink/1.2.7.3.html.

  20. The numbers reported here are for all available widgets, including non-interactable widgets (e.g., labels, pictures) and invisible widgets (e.g., layout panels, tab panels).

  21. The prefix “G” indicates that a component is a GUITAR abstract class.

  22. http://jenkins-ci.org.

  23. http://cobertura.sourceforge.net.

References

  • Alfaro, L.D., Henzinger, T.A.: Interface theories for component-based design. In: Proceedings of the First International Workshop on Embedded Software (EMSOFT ’01), pp. 148–165. Springer, London (2001)

    Google Scholar 

  • Amalfitano, D., Fasolino, A.R., Tramontana, P.: A GUI crawling-based technique for Android mobile application testing. In: Proceedings of the IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW ’11), pp. 252–261. IEEE Comput. Soc., Los Alamitos (2011)

    Chapter  Google Scholar 

  • Artzi, S., Dolby, J., Jensen, S.H., Møller, A., Tip, F.: A framework for automated testing of javascript web applications. In: Proceedings of the 33rd International Conference on Software Engineering (ICSE ’11), pp. 571–580. ACM, New York (2011)

    Google Scholar 

  • Baresi, L., Young, M.: Test oracles. Technical report CIS-TR-01-02, University of Oregon, Dept. of Computer and Information Science, Eugene, Oregon, USA (2001)

  • Belli, F.: Finite-state testing and analysis of graphical user interfaces. In: Proceedings of the 12th International Symposium on Software Reliability Engineering (ISSR ’01), p. 34. IEEE Comput. Soc., Washington (2001)

    Chapter  Google Scholar 

  • Brooks, P.A., Memon, A.M.: Automated GUI testing guided by usage profiles. In: Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering (ASE ’07), pp. 333–342. ACM, New York (2007)

    Chapter  Google Scholar 

  • Brooks, P., Robinson, B., Memon, A.M.: An initial characterization of industrial graphical user interface systems. In: Proceedings of the 2nd IEEE International Conference on Software Testing, Verification and Validation (ICST 2009). IEEE Computer Society, Washington (2009)

    Google Scholar 

  • Cadar, C., Dunbar, D., Engler, D.: KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI ’08), pp. 209–224. USENIX Association, Berkeley (2008)

    Google Scholar 

  • Chang, T.H., Yeh, T., Miller, R.C.: GUI testing using computer vision. In: Conference on Human Factors in Computing Systems, pp. 1535–1544 (2010)

    Google Scholar 

  • Chen, J., Subramaniam, S.: A GUI environment to manipulate FSMs for testing GUI-based applications in Java. In: Conference on System Sciences, vol. 9, p. 9061 (2001)

    Google Scholar 

  • Chen, W.K., Tsai, T.H., Chao, H.H.: Integration of specification-based and CR-based approaches for GUI testing. In: Conference on Advanced Information Networking and Applications, pp. 967–972 (2005)

    Google Scholar 

  • Cunha, M., Paiva, A., Ferreira, H., Abreu, R.: PETTool: a pattern-based GUI testing tool. In: International Conference on Software Technology and Engineering, pp. 202–206 (2010)

    Google Scholar 

  • Draheim, D., Lutteroth, C., Weber, G.: A source code independent reverse engineering tool for dynamic web sites. In: Proceedings of the Ninth European Conference on Software Maintenance and Reengineering (CSMR ’05), pp. 168–177. IEEE Comput. Soc., Washington (2005)

    Chapter  Google Scholar 

  • Ganov, S., Kilmar, C., Khurshid, S., Perry, D.: Test generation for graphical user interfaces based on symbolic execution. In: Proceedings of the International Workshop on Automation of Software Test (2008)

    Google Scholar 

  • Grechanik, M., Xie, Q., Fu, C.: Creating GUI testing tools using accessibility technologies. In: Conference on Software Testing, Verification, and Validation, pp. 243–250 (2009)

    Google Scholar 

  • Hellmann, T.D., Hosseini Khayat, A., Maurer, F.: Supporting test-driven development of graphical user interfaces using agile interaction design. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, pp. 444–447 (2010)

    Chapter  Google Scholar 

  • Huang, S., Cohen, M.B., Memon, A.M.: Repairing GUI test suites using a genetic algorithm. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation (ICST ’10), pp. 245–254. IEEE Comput. Soc., Washington (2010)

    Chapter  Google Scholar 

  • Jääskeläinen, A., Katara, M., Kervinen, A., Maunumaa, M., Pääkkönen, T., Takala, T., Virtanen, H.: Automatic GUI test generation for smartphone applications—an evaluation. In: Proceedings of the Software Engineering in Practice Track of the 31st International Conference on Software Engineering (ICSE 2009), pp. 112–122 (companion volume). IEEE Computer Society, Los Alamitos (2009)

    Google Scholar 

  • McMaster, S., Memon, A.M.: An extensible heuristic-based framework for GUI test case maintenance. In: Proceedings of the First International Workshop on TESTing Techniques & Experimentation Benchmarks for Event-Driven Software (TESTBEDS ’09). IEEE Computer Society, Washington (2009)

    Google Scholar 

  • Memon, A.M.: An event-flow model of GUI-based applications for testing. Softw. Test. Verif. Reliab. 17, 137–157 (2007)

    Article  Google Scholar 

  • Memon, A.M., Nguyen, B.N.: Advances in automated model-based system testing of software applications with a GUI front-end. In: Zelkowitz, M.V. (ed.) Advances in Computers, vol. 80, pp. 121–162. Academic Press, San Diego (2010)

    Google Scholar 

  • Memon, A.M., Xie, Q.: Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software. IEEE Trans. Softw. Eng. 31(10), 884–896 (2005)

    Article  Google Scholar 

  • Memon, A.M., Banerjee, I., Nagarajan, A.: GUI ripping: reverse engineering of graphical user interfaces for testing. In: Proceedings of the 10th Working Conference on Reverse Engineering (WCRE ’03), p. 260. IEEE Computer Society, Washington (2003)

    Google Scholar 

  • Memon, A.M., Pollack, M.E., Soffa, M.L.: Hierarchical GUI test case generation using automated planning. IEEE Trans. Softw. Eng. 27(2), 144–155 (2001a)

    Article  Google Scholar 

  • Memon, A.M., Soffa, M.L., Pollack, M.E.: Coverage criteria for GUI testing. In: Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering, vol. 26, pp. 256–267. ACM, New York (2001b)

    Google Scholar 

  • Mesbah, A., van Deursen, A.: Invariant-based automatic testing of AJAX user interfaces. In: Proceedings of the 31st International Conference on Software Engineering (ICSE ’09), pp. 210–220. IEEE Computer Society, Washington (2009)

    Google Scholar 

  • Myers, B.A.: User interface software tools. ACM Trans. Comput.-Hum. Interact. 2(1), 64–103 (1995)

    Article  Google Scholar 

  • Nguyen, D.H., Strooper, P., Suess, J.G.: Model-based testing of multiple GUI variants using the GUI test generator. In: Workshop on Automation of Software Test, pp. 24–30 (2010)

    Chapter  Google Scholar 

  • Paiva, A.C.R., Faria, J.C.P., Mendes, P.M.C.: Reverse engineered formal models for GUI testing. In: Proc. of Conf. on Formal Methods for Industrial Critical Systems, 4916(1), pp. 218–233 (2008)

    Chapter  Google Scholar 

  • Robinson, B., Francis, P., Ekdahl, F.: A defect-driven process for software quality improvement. In: Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’08), pp. 333–335. ACM, New York (2008). doi:10.1145/1414004.1414072

    Chapter  Google Scholar 

  • Ruiz, A., Price, Y.W.: GUI testing made easy. In: Proceedings of the Testing: Academic & Industrial Conference—Practice and Research Techniques, pp. 99–103. IEEE Computer Society, Washington (2008)

    Google Scholar 

  • Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S.: Designing the User Interface: Strategies for Effective Human–Computer Interaction, 5th edn. Addison-Wesley, Reading (2009)

    Google Scholar 

  • Silva, J.L., Campos, J.C., Paiva, A.C.R.: Model-based user interface testing with spec explorer and ConcurTaskTrees. Electron. Notes Theor. Comput. Sci. 208, 77–93 (2008)

    Article  Google Scholar 

  • Silva, J.C., Saraiva, J., Campos, J.C.: A generic library for GUI reasoning and testing. In: ACM Symposium on Applied Computing, pp. 121–128 (2009)

    Google Scholar 

  • Staiger, S.: Static analysis of programs with graphical user interface. In: 11th European Conference on Software Maintenance and Reengineering (CSMR ’07), pp. 252–264 (2007)

    Chapter  Google Scholar 

  • Ural, H., Yang, B.: A test sequence selection method for protocol testing. IEEE Trans. Commun. 39(4), 514–523 (1991)

    Article  Google Scholar 

  • Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L.: Model-based testing of object-oriented reactive systems with Spec Explorer. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing, Chap.: Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer, pp. 39–76. Springer, Berlin (2008)

    Chapter  Google Scholar 

  • Vieira, M., Leduc, J., Hasling, B., Subramanyan, R., Kazmeier, J.: Automation of GUI testing using a model-driven approach. In: Conference on Software Engineering, pp. 9–14 (2006)

    Google Scholar 

  • Xie, Q., Memon, A.M.M.: Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Eng. Methodol. 16(1), 4 (2007)

    Article  Google Scholar 

  • Xie, Q., Memon, A.M.: Using a pilot study to derive a GUI model for automated testing. ACM Trans. on Softw. Eng. and Method. (2008)

  • Yuan, X., Memon, A.M.: Generating event sequence-based test cases using GUI runtime state feedback. IEEE Trans. Softw. Eng. 36, 81–95 (2010)

    Article  Google Scholar 

  • Yuan, X., Cohen, M.B., Memon, A.M.: GUI interaction testing: incorporating event context. IEEE Trans. Softw. Eng. 37(4), 559–574 (2011)

    Article  Google Scholar 

  • Zhang, S., Saff, D., Bu, Y., Ernst, M.D.: Combined static and dynamic automated test generation. In: Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA ’11), pp. 353–363. ACM, New York (2011)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Maryland, College Park, MD, 20742, USA

    Bao N. Nguyen, Bryan Robbins, Ishan Banerjee & Atif Memon

Authors
  1. Bao N. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bryan Robbins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ishan Banerjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Atif Memon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bao N. Nguyen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nguyen, B.N., Robbins, B., Banerjee, I. et al. GUITAR: an innovative tool for automated testing of GUI-driven software. Autom Softw Eng 21, 65–105 (2014). https://doi.org/10.1007/s10515-013-0128-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-013-0128-9

Keywords

Search

Navigation