Tsung-Hsiang Chang
ABSTRACT
Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically. We show that a variety of GUI behavior can be tested using this approach. Also, we show how this approach can facilitate good testing practices such as unit testing, regression testing, and test-driven development.
- Autoit. http://www.autoitscript.com/autoit3/, 1999.Google Scholar
- M. Boshernitsan, S.L. Graham, and M.A. Hearst. Aligning development tools with the way programmers think about code changes. In CHI '07, pages 567--576, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- HP Mercury Interactive. Winrunner. http://www.winrunner.com.Google Scholar
- D.J. Kasik and H.G. George. Toward automatic generation of novice user test scripts. In CHI '96, pages 244--251, New York, NY, USA, 1996. ACM. Google ScholarDigital Library
- A.J. Koand, B.A. Myers. Designing the why line: a debugging interface for asking questions about program behavior. In CHI '04, pages 151--158, New York, NY, USA, 2004. ACM. Google ScholarDigital Library
- A.J. Koand, B.A. Myers. Finding causes of program output with the java why line. In CHI '09, pages 1569--1578, New York, NY, USA, 2009. ACM. Google ScholarDigital Library
- G. Leshed, E.M. Haber, T. Matthews, and T. Lau. Coscripter: automating & sharing how-to knowledge in the enterprise. In CHI '08, pages 1719--1728, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- G. Little, T.A. Lau, A. Cypher, J. Lin, E.M. Haber, and E. Kandogan. Koala: capture, share, automate, personalize business processes on the web. In CHI '07, pages 943--946, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- A. Memon. GUI testing: pitfalls and process. Computer, 35(8):87--88, Aug 2002. Google ScholarDigital Library
- A. Memon, I. Banerjee, and A. Nagarajan. GUI ripping: reverse engineering of graphical user interfaces for testing. In Reverse Engineering, 2003. WCRE 2003. Proceedings. 10th Working Conference on, pages 260--269, Nov. 2003. Google ScholarDigital Library
- T. Ostrand, A. Anodide, H. Foster, and T. Goradia. A visual test development environment for GUI systems. SIGSOFT Softw. Eng. Notes, 23(2):82--92, 1998. Google ScholarDigital Library
- G. Singh and Z. Cuie. Sage: creating reusable, modularized interactive behaviors by demonstration. In CHI '94, pages 297--298, New York, NY, USA, 1994. ACM. Google ScholarDigital Library
- SOFTWARERESEARCHINC.Capbak.http://soft.com, 1999.Google Scholar
- R. St. Amant, H. Lieberman, R. Potter, and L. Zettlemoyer. Programming by example: visual generalization in programming by example. Commun. ACM, 43(3):107--114, 2000. Google ScholarDigital Library
- N. Subrahmaniyan, L. Beckwith, V. Grigoreanu, M. Burnett, S. Wiedenbeck, V. Narayanan, K. Bucht, R. Drummond, and X. Fern. Testing vs. code inspection vs. what else?: male and female end users' debugging strategies. In CHI '08, pages 617--626, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- E.M. Wilcox, J.W. Atwood, M.M. Burnett, J.J. Cadiz, and C.R. Cook. Does continuous visual feedback aid debugging in direct-manipulation programming systems? In CHI '97, pages 258--265, New York, NY, USA, 1997. ACM. Google ScholarDigital Library
- Q. Xie and A.M. Memon. Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Eng. Methodol., 16(1):4, 2007. Google ScholarDigital Library
- T. Yeh, T.-H. Chang, and R.C. Miller. Sikuli: Using GUI screenshots for search and automation. In UIST '09, pages 183--192. ACM, 2009. Google ScholarDigital Library
- L.S. Zettlemoyer and R. St. Amant. A visual medium for programmatic control of interactive applications. In CHI '99, pages 199--206, New York, NY, USA, 1999. ACM. Google ScholarDigital Library
Index Terms
- GUI testing using computer vision
Recommendations
Coverage criteria for GUI testing
ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineeringA widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Coverage criteria for GUI testing
A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
GUI Interaction Testing: Incorporating Event Context
Graphical user interfaces (GUIs), due to their event-driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing, it is important to “adequately cover” this interaction space. In this paper, we ...
Comments