Elsevier

Advances in Computers

Volume 101, 2016, Pages 193-237
Advances in Computers

Chapter Five - Approaches and Tools for Automated End-to-End Web Testing

https://doi.org/10.1016/bs.adcom.2015.11.007Get rights and content

Abstract

The importance of test automation in web engineering comes from the widespread use of web applications and the associated demand for code quality. Test automation is considered crucial for delivering the quality levels expected by users, since it can save a lot of time in testing and it helps developers to release web applications with fewer defects. The main advantage of test automation comes from fast, unattended execution of a set of tests after some changes have been made to a web application. Moreover, modern web applications adopt a multitier architecture where the implementation is scattered across different layers and run on different machines. For this reason, end-to-end testing techniques are required to test the overall behavior of web applications.

In the last years, several approaches have been proposed for automated end-to-end web testing and the choice among them depends on a number of factors, including the tools used for web testing and the costs associated with their adoption. They can be classified using two main criteria: the first concerns how test cases are developed (ie, Capture-Replay and Programmable approaches), while, the second concerns how test cases localize the web elements to interact with (ie, Coordinates-based, DOM-based, and Visual approaches), that is what kind of locators are used for selecting the target GUI components.

For developers and project managers it is not easy to select the most suitable automated end-to-end web testing approach for their needs among the existing ones. This chapter provides a comprehensive overview of the automated end-to-end web testing approaches and summarizes the findings of a long term research project aimed at empirically investigating their strengths and weaknesses.

Introduction

Web applications are key assets of our society. A considerable slice of modern software consists of web applications executed in the user's web browser, running on computers or smartphones. The web has a significant impact on all aspects of our society and in the last years has changed the life of billions of people. Associations, enterprizes, governmental organizations, companies, scientific groups use the web as a powerful and convenient way to promote activities/products and carry out their core business. People daily use online services as source of information, means of communication, source of entertainment, and venue for commerce. In a sentence, web applications pervade our life, being crucial for a multitude of economic, social and educational activities.

The importance of the web in our lives stresses the quality with which these applications are developed and maintained [1]. End-to-end web testing is one of the main approaches for assuring the quality of web application [2]. The goal of end-to-end web testing is exercising the web application under test as a whole to detect as many failures as possible, where a failure can be considered as a deviation from the expected behavior. In many software projects, end-to-end web testing is neglected because of time or cost constraints. However, the impact of failures in a web application may be very serious, ranging from simple inconvenience (eg, malfunction and so users’ dissatisfaction), economic problems (eg, interruption of business), up to catastrophic impacts.

The simplest solution is to manually interact with the web application under development to see if it behaves as expected. Unfortunately, this practice is error prone, time consuming, and ultimately not very effective. For this reason, most teams automate manual web testing by means of automated testing tools. The process contains a first manual step: producing the test code able to instrument the web application. Test code provides input data, operates on GUI components, and retrieves information to be compared with oracles (eg, using assertions). The main benefit of test automation comes from the fast and unattended execution of a test suite after some changes have been made to the web application under test (ie, for regression purposes).

End-to-end testing of web applications is a type of black box testing based on the concept of test scenario, that is a sequence of steps/actions performed on the web application (eg, insert username, insert password, click the login button). One or more test cases can be derived from a single test scenario by specifying the actual data to use in each step (eg, username=John.Doe) and the expected results (ie, defining the assertions). The execution of each test case can be automated by implementing a test script following any of the existing approaches (eg, Programmable and DOM-based localization). The choice among the various approaches depends on a number of factors, including the technology used by the web applications and the tools (if any) used for web testing.

Broadly speaking, there are two main orthogonal criteria [3] to classify the approaches to end-to-end web testing that are related to: (1) test scripts implementation and (2) web page elements localization. Fig. 1 shows a classification grid based on these two criteria that can be applied to existing tools. For what concerns the first criterion, we can find two main approaches [4]:

  • Capture-Replay (C&R) Web Testing consists of recording the actions performed by the tester on the web application GUI and generating a test script that repeats such actions for automated, unattended reexecution.

  • Programmable Web Testing aims at unifying web testing with traditional testing, where test scripts are themselves software artifacts that developers write, with the help of specific testing frameworks. Such frameworks allow developers to program the interactions with a web page and its elements, so that test scripts can, for instance, automatically fill-in and submit forms or click on hyper-links.

An automated end-to-end test case interacts with several web page elements such as links, buttons, and input fields, and different methods can be used to locate them. Thus, concerning the second criterion, we can find three different cases [3]:

  • Coordinate-based localization: the tools implementing this approach just record the screen coordinates of the web page elements and then use this information to locate the elements during test case replay. This approach is nowadays considered obsolete, because it produces test scripts that are extremely fragile. Hence, it is not considered any further in this work.

  • DOM-based localization: the tools implementing this approach (eg, Selenium IDE1 and Selenium WebDriver2 ) locate the web page elements using the information contained in the Document Object Model (DOM) and, usually, provide several ways to locate web page elements. For instance, Selenium WebDriver is able to locate a web page element using: (1) the values of attributes id, name, and class; (2) the tag name of the element; (3) the text string shown in the hyperlink, for anchor elements; (4) CSS and (5) XPath expressions. Not all these locators are applicable to any arbitrary web element; eg, locator (1) can be used only if the target element has a unique value of attribute id, name, or class in the entire web page; locator (2) can be used if there is only one element with the chosen tag name in the whole page; and, locator (3) can be used only for links uniquely identified by their text string. On the other hand, XPath/CSS expressions can always be used. In fact, as a baseline, the unique path from root to target element in the DOM tree can always be turned into an XPath/CSS locator that uniquely identifies the element.

  • Visual localization: the tools implementing this approach have emerged recently. They make use of image recognition techniques to identify and control GUI components. The tool Sikuli IDE3 and Sikuli API4 belong to this category.

For test developers and project managers it is not easy to select the most suitable automated end-to-end web testing approach for their needs among the existing ones. This chapter gives a comprehensive overview of the automated end-to-end web testing approaches and summarizes the findings of a long term research project [[3], [4], [1], [6]] aimed at empirically investigating their strengths and weaknesses.

The main contribution of this chapter is twofold:

  • Providing the reader with a complete overview of the most relevant approaches in the context of the automated end-to-end web testing.

  • Analyzing such approaches in order to point out their strengths and weaknesses.

The considered approaches have been analyzed by means of a series of empirical studies [[3], [4]] in which we developed and compared several test suites for six open source web applications. In particular:

  • (1)

    Concerning test case development and maintenance, we empirically investigated the trade-off between capture-replay (C&R) and programmable web testing [4]. In addition to validating our initial hypothesis, that C&R test cases are cheaper to write from scratch than programmable test cases, but are also more expensive to maintain during software evolution, we have determined the number of software releases after which the savings of programmable test cases overcome the costs initially paid for their development (see Fig. 2).

    We have instantiated our empirical analysis for two state-of-the-practice tools, Selenium IDE (ie, C&R), and Selenium WebDriver (ie, programmable).

  • (2)

    Concerning how test cases localize the web element to interact with, we have evaluated and compared the visual and DOM-based approaches [3] considering: the robustness of locators, the initial test suite development effort, the test suite evolution cost, and the test suite execution time. Our empirical assessment of the robustness of locators is quite general and tool independent, while the developers’ effort for initial test suite development and the effort for test suite evolution were measured with reference to specific implementations of the two approaches. We have instantiated such analysis for two tools, Sikuli API and Selenium WebDriver, both adopting the programmable approach but differing in the way they localize the web elements to interact with during the execution of the test cases. Indeed, Sikuli API adopts the visual approach, thus using images representing portions of the web pages, while Selenium WebDriver employs the DOM-based approach, thus relying on the HTML structure. Since visual tools are known to be computational demanding, we also measured and compared the test suite execution time.

The findings reported in this chapter provide practical guidelines for developers who want to make an informed decision among the available approaches and who want to understand which of them could fit more or less well for a specific web development context.

The chapter is organized as follows: Sections 2 and 3 provide an overview on the main classical approaches to automated end-to-end web testing and report several examples of tools instantiating such approaches. Specifically, these sections describes how the test cases development approaches (ie, capture-replay and programmable) can be combined with the DOM-based and the visual localization approaches. Section 4 describes how the evolution of the web application impacts on the test cases created by following each approach. Section 5 summarizes and discusses the results of the empirical studies we conducted to analyze the strengths and the weaknesses of various approaches for automated end-to-end web testing. Section 6 analyses some tools and techniques that have been recently proposed, overcoming the limitations of the classical approaches to automated end-to-end web testing. In particular Section 6.1 provides some example of tools/techniques that go beyond the simple adoption of one approach, ie, solutions that are able to combine more approaches at the same time (eg, Visual + DOM based or C&R + Programmable). Then, Section 6.2 analyses a set of techniques that have been proposed in the literature in order to solve specific problems in the context of automated end-to-end web testing (robustness, test case repair upon software evolution, page object creation, and migration between approaches). Section 7 concludes the chapter.

Section snippets

Capture-Replay Web Testing

Capture-Replay web testing is based on the usage of capture/replay automated tools. These tools have been developed to validate interactive applications (GUI or web applications). Using a capture/replay tool, a software tester can run a web application and record the entire interaction session. The tool (semi-) automatically records all the user's events in the navigated web pages, such as the key presses and mouse clicks, in a script, allowing a session to be rerun automatically without

Programmable Web Testing

Programmable web testing is based on manual creation of a test script. Web test scripts can be written using ad-hoc languages and frameworks or general purpose programming languages (such as Java and Ruby) with the aid of specific libraries able to play the role of the browser. Usually, these libraries extend the programming language with user friendly APIs, providing commands to, eg, click a button, fill a field, and submit a form. Test scripts are completed with assertions (eg, JUnit

Test Case Evolution

The main benefits of adopting test automation are the possibility of: (1) executing the test cases more often, (2) finding bugs in the early stage of development, and (3) reusing test code across successive releases of the web application under test (ie, for regression testing). The main cost associated with test automation is related to the fragility of test cases: a fragile test is a test that is broken when the web application under test is slightly modified. Specifically, when a web

Analysis of the Approaches

This section summarizes a series of empirical studies [[3], [4], [1]] we conducted to analyze the strengths and the weaknesses of various approaches to automated end-to-end web testing. We analyzed the four main approaches currently adopted in the context of automated functional web testing. In particular, for what concerns the test scripts implementation, we compared C&R and Programmable approaches, while concerning the web page elements localization we compared Visual and DOM-based approaches.

Overcoming the Limitations of the Existing Approaches

In the previous sections we provided an overview of the existing, consolidated approaches to automated end-to-end web testing and some examples of tools implementing them. In this section we discuss recent research works and existing commercial tools that try to overcome their main limitations.

Conclusions

In this chapter we have provided an overview of the most relevant approaches and tools to automated end-to-end web testing. First, for each approach we have given a detailed description based on existing implementations; second, we have deeply analyzed their strengths and weaknesses by discussing the results of a series of empirical studies. Third, we have described some recent techniques and tools that try to overcome the limitations of the existing approaches by combining them into hybrid

Maurizio Leotta is a research fellow at the University of Genova, Italy. He received his PhD degree in Computer Science from the same University, in 2015, with the thesis “Automated Web Testing: Analysis and Maintenance Effort Reduction.” He is author or coauthor of more than 30 research papers published in international conferences and workshops. His current research interests are in Software Engineering, with a particular focus on the following themes: Web Application Testing, Functional

References (24)

  • P. Tonella et al.

    Recent advances in web testing

    Adv. Comput.

    (2014)
  • F. Ricca et al.

    Web application transformations based on rewrite rules

    Inf. Softw. Technol.

    (2002)
  • P. Montoto et al.

    Automated browsing in AJAX websites

    Data Knowl. Eng.

    (2011)
  • F. Ricca et al.

    Detecting anomaly and failure in web applications

    IEEE Multimed.

    (2006)
  • F. Ricca et al.

    Analysis and testing of web applications

  • M. Leotta et al.

    Visual vs. DOM-based web locators: an empirical study

  • M. Leotta et al.

    Capture-replay vs. programmable web testing: an empirical assessment during test case evolution

  • M. Leotta et al.

    Improving test suites maintainability with the Page Object pattern: an industrial case study

  • M. Leotta et al.

    Comparing the maintainability of Selenium WebDriver test suites employing different locators: a case study

  • A. van Deursen

    Beyond page objects: testing web applications with state objects

    ACM Queue

    (2015)
  • M. Mirzaaghaei

    Automatic test suite evolution

  • R. Yandrapally et al.

    Robust test automation using contextual clues

  • Cited by (62)

    • An automated model-based approach to repair test suites of evolving web applications

      2021, Journal of Systems and Software
      Citation Excerpt :

      The test scripts for the industrial case study are written in Selenium Web driver. For open-source applications, we used the six case studies that are widely used in web testing research (Leotta et al., 2016), primarily in the studies evaluating test case repair strategies (Leotta et al., 2015, 2014, 2013). According to a recent literature review on test breakage prevention and repair techniques (Imtiaz et al., 2019), these six open-source web applications are the most commonly used subject applications in literature and therefore can be considered as a benchmark for empirical evaluation.

    • ATF - An end-to-end testing framework: experience report

      2023, ACM International Conference Proceeding Series
    • Cytestion: Automated GUI Testing for Web Applications

      2023, ACM International Conference Proceeding Series
    View all citing articles on Scopus

    Maurizio Leotta is a research fellow at the University of Genova, Italy. He received his PhD degree in Computer Science from the same University, in 2015, with the thesis “Automated Web Testing: Analysis and Maintenance Effort Reduction.” He is author or coauthor of more than 30 research papers published in international conferences and workshops. His current research interests are in Software Engineering, with a particular focus on the following themes: Web Application Testing, Functional Testing Automation, Business Process Modelling, Empirical Software Engineering, Model-Driven Software Engineering.

    Diego Clerissi is a PhD student in Computer Science at the University of Genova, Italy. In 2015 he received his master degree from the same University, with the thesis: “Test Cases Generation for Web Applications from Requirements Specification: Preliminary Results.” At the time of writing he is coauthor of 10 research papers published in international conferences and workshops. His research interests are in Software Engineering, Model-Based Testing, Software Testing, Web Applications, System Modeling.

    Filippo Ricca is an associate professor at the University of Genova, Italy. He received his PhD degree in Computer Science from the same University, in 2003, with the thesis: “Analysis, Testing and Re-structuring of Web Applications.” In 2011 he was awarded the ICSE 2001 MIP (Most Influential Paper) award, for his paper: “Analysis and Testing of Web Applications.” He is author or coauthor of more than 100 research papers published in international journals and conferences/workshops. He was Program Chair of CSMR/WCRE 2014, CSMR 2013, ICPC 2011, and WSE 2008. Among the others, he served in the program committees of the following conferences: ICSM, ICST, SCAM, CSMR, WCRE, and ESEM. From 1999 to 2006, he worked with the Software Engineering group at ITC-irst (now FBK-irst), Trento, Italy. During this time he was part of the team that worked on Reverse engineering, Re-engineering, and Software Testing. His current research interests include: Software modeling, Reverse engineering, Empirical studies in Software Engineering, Web applications, and Software Testing. The research is mainly conducted through empirical methods such as case studies, controlled experiments, and surveys.

    Paolo Tonella is head of the Software Engineering Research Unit at Fondazione Bruno Kessler (FBK), in Trento, Italy. He received his PhD degree in Software Engineering from the University of Padova, in 1999, with the thesis: “Code Analysis in Support to Software Maintenance.” In 2011 he was awarded the ICSE 2001 MIP (Most Influential Paper) award, for his paper: “Analysis and Testing of Web Applications.” He is the author of “Reverse Engineering of Object Oriented Code,” Springer, 2005. He participated in several industrial and EU projects on software analysis and testing. His current research interests include code analysis, web and object oriented testing, search-based test case generation.

    View full text