Keywords

1 Introduction

Cloud computing has different characteristics which are varied based on the area of the providing service, but the most important ones can be categorized as follows.

  • On-demand service: Customer can request for resources on a flexible short-term basis and get charged based on the utility of resources [3].

  • Rapid elasticity: Customer can rapidly request for more resources or release unnecessary resources as needed [4].

  • Virtualization: Customer can enjoy the portability and versatility of cloud when the lower level hardware is abstracted [4].

  • High reliability: Customer can execute time-consuming operations on the cloud because the cloud is built to be fault tolerant [5].

The cloud services are usually classified into three major sections as infrastructure, platform and software, however, Testing-as-a-Service (TaaS) is recently known as a new and demanding type of service model.

  • Infrastructure-as-a-Service (IaaS): Customer can acquire various computing hardware, including data storage, network devices, and processing power [1].

  • Platform-as-a-Service (PaaS): Customer can access a set of developer environments, including operating systems, database, and web server [4].

  • Software-as-a-Service (SaaS): Customer can access software applications through a client interface such as the web browser without the complexity of installations [4].

  • Testing-as-a-Service (TaaS): Customer can access test simulation environments and monitor complex program behaviors continuously [6].

Much research being conducted on cloud testing and most notable published studies include Testing Scope in Cloud, [7], cloud testing review [8], Integrated TaaS Platform for Mobile Development [9], A Systematic Mapping Study of Empirical Studies on Software Cloud Testing Methods [10], Research on Testing Software for Rapid Cloud Deployment [11], Cloud API Testing [12].

Cloud testing can have multiple interpretations: testing of cloud applications or testing as a cloud service. In general, Service Level Agreement (SLA) is a contract between cloud customer and cloud provider which describes the required quality of service (QoS) and the penalties for QoS violations. Adequate SLA monitoring can be of interest to both end users and cloud providers [13]. From the end user perspective, it is helpful to know which the best eligible cloud provider is, based on systematic testing of attributes such as reliability, performance, and security. Many researchers have investigated in this area. For example, Wagle et al. [14] proposed an evaluation model for ranking commercially available cloud providers using an ordered performance heat map. From the cloud provider perspective, it is important to monitor the health of the cloud system and to avoid the penalties for QoS violations. However, according to [15] testing of the cloud is a non-trivial task that requires continuous measurements of cloud application status and effective monitoring across multiple layers of the cloud stack.

As modern web applications are becoming more complex, traditional in-house testing also becomes insufficient in several ways. In-house testing facilities need to be carefully configured and maintained, and there is a significant amount of wastage when these facilities are in the idle state. Furthermore, as discussed in [4] in-house testing facilities are usually not powerful enough to simulate real-world concurrent user traffic. Due to the increasing awareness of testing as a cloud service, many businesses [16] have started to consider the trade-offs between keeping their existing testing facilities and migrating testing activities to the cloud. In fact, using the cloud as a testing middleware according to [2] promotes multi-scenario testing evaluation due to distributed testing and offers a large-scale simulation of real-time user interactions. In order to further understand the benefits of testing as a cloud service, it is useful to explore how testing environments are constructed in the cloud.

2 Cloud Services and Testing

This section describes some published methodologies on testing of cloud and testing as a cloud service, where the methodologies are categorized based on the type of testing, the type of cloud service, and the evaluation method. Table 1 lists the reference numbers of research papers according to the type of cloud service and the type of testing and Fig. 1 illustrates the breakdown of research papers according to the evaluation methods.

Table 1. Distribution according to cloud services and testing.
Full size table
Fig. 1.
figure 1

Distribution of researches according to evaluation method

Full size image

2.1 Types of Testing

Functionality.

Testing: The key focus of functionality testing is to reveal deviations between software’s intended and actual functionality. Gao et al. [17] developed a cloud-based testing as a service (CTaaS) system with a UI-layer, test space layer, and TaaS layer. Their system includes a pool of GUI-based test scripts from which cloud customers can select and obtain test simulation results after connecting to a GUI- based tool, such as Selenium. The approach that was suggested by Rosiello et al. [18] combines black box testing and the technique of fault injection. The authors introduce faults into the system under test by killing selected server threads and observing the output, e.g. the number of successful server responses, to see if there are any inconsistencies in system behavior.

Performance Testing.

The key focus of performance testing is to determine software’s responsiveness and effectiveness with numerical measurements. Nasiri et al. [19] described a multi-component framework consisting of an application analyzer to find the testable region in source code, a test case generator using SoapUI, and a cloud-based test executor. With this framework, they collected the mean execution times of activities in testing and discovered that the testbed startup time can be reduced by simultaneously initializing the system under test (SUT) master node and the test system machine.

A cloud-based testing as a service (CTaaS) system introduced in [17], includes a UI-layer, test space layer, and TaaS layer. The CTaaS includes a pool of performance test scripts, automatic test script generation using workflow graphs, and SaaS performance metrics collection using Amazon’s EC2 CloudWatch APIs. The study was further supported by [20] declarative testing environment consisting of Crawl, a domain-specific language for defining performance test scenarios, and Crawler, a Java-based cloud engine for executing test scenarios and collecting the results. Others [13] created multi-module framework consisting of one design time module and four runtime modules. One of the runtime modules is responsible for computing response time, throughput, and reliability in order to detect SLA violations. The further study [21] of a mobile application testing framework that takes Android application as input, records application execution using a crawler, and output performance metrics of response time, throughput, and network latency.

A model-based approach was suggested by [22] for performance testing. The authors use a sequential action model to define the cloud services to be tested and a load model to define parameters such as test execution duration and number of simulated users. To this end, the further work [23] describes a framework to conduct performance testing on cloud applications. Their framework is based on identifying operational modes and actors, calculating occurrence probability of common operations, and measuring performance in terms of transaction time and page load time. Furthermore, Ghosh et al. [24] proposed three stochastic models (resource provisioning decision model, VM provisioning model, and run-time model) to describe the inner details of a cloud service. Based on these models, the authors also introduced formulas to calculate QoS metrics of job rejection probability and mean decision delay, which are indicators of system availability and response time.

Elasticity and Scalability Testing.

The key focus of elasticity and scalability testing is to determine the cloud’s adapting ability when the demand for resources changes. A perfectly elastic cloud can instantaneously scale resources up or down according to demand. However, as described in [25] in reality, there is always a delay between the time when a load change is detected and the time when resource configuration is changed accordingly. Further work on real-time elasticity testing by Albonico et al. [26] includes a framework with three different elasticity states (i) a scaling in the state when a resource is released, (ii) a scaling out state when a resource is acquired, and (iii) a ready state. For each elasticity state, they executed 2500 operations per second for a fixed period of time and recorded the number of operations that are successful. The authors used these recordings to identify problems during each elasticity state. Further research by Tsai et al. [27] reveals a framework that uses feature selection algorithms to identify bottleneck of an application and uses association rule to identify significant relationships between scalability and various parameters, such as a number of concurrent users.

Security Testing.

The key focus of security testing is to reveal security leaks that make the software vulnerable to attacks. Cotroneo et al. [28] described a framework to identify the root cause of security alerts through a conceptual clustering approach. Their framework is valuable to testing because large volumes of security issues can be detected and classified automatically. Furthermore, an identity-based cloud data integrity checking protocol [29] that consists of six algorithms and can be used for auditing variable-sized data blocks. A framework described by [30] consists of a front-end module that uses the Model-View-Controller pattern, a test environment module that prepares tools and virtual machines, and a testing module that handles test scripts. To achieve security testing, the authors integrated Metasploit, a testing tool that offers security vulnerability scanning, into the test environment module.

A multi-component framework described by [31] consisted a test scheduler for allocating resources, a test controller for checking the status of security scanners, and several other components. The authors mentioned a set of security scanners, including IBM AppScan and HP WebInspect that can be used to identify vulnerabilities.

Automatic Testing.

The automation of test case generation is another widely explored aspect of cloud testing. A framework proposed by [32] where semantic information in the form of Web Service Description Language (WSDL) is generated from source code and comments, event sequence graphs are generated from WSDL information, and test cases are generated from event sequence graphs. Further research conducted by [33] describes an architecture called Expertus to automate testing of large-scale applications in the cloud. Expertus uses two templates to generate and modify cloud resources and a multi-stage, XML-based code generator to systematically generate complex test scenarios. An interesting research work by [34] reveals layered parameterized tests that use layered symbolic execution (LSE) to dynamically automate test script generation. The LSE algorithm explores the path- based execution tree of an application and obtains inputs for testing by negating constraints in the path conditions. Based on cloud computing, [59] developed a multi-layered model of a software online testing platform which integrates IaaS and SaaS platforms as an automatic self-help service portal for users and operation maintenance portal for administrators.

2.2 Testing Cloud Service

Infrastructure as a Service.

There are multiple articles published in this area each with its own unique and significant contributions. A declarative testing environment that can execute tests in multiple IaaS clouds was discussed by [20]. Others including [35] suggested using a third-party auditor to detect SLA violations of virtual machine CPU speed. The third-party auditor (TPA) must have its own timing functions and minimal communication overhead. Hence SLA violations can be detected by comparing the execution times on the cloud provider’s virtual machine and on the TPA’s virtual machine. Yet [29] describes a protocol for data integrity checking on IaaS cloud. The advantage of IaaS over traditional server was demonstrated [23] by deploying the same application on both infrastructures and comparing their performance under high load conditions. This was supported by a general description of IaaS cloud using three stochastic models [24], where some areas more than one of these cloud services are needed as in the autonomous neural network-based 3D positioning systems [55] which are very useful in rescue operations and need very reliable software and testing services.

Platform as a Service.

Model-based approaches including [22, 34] that are used for testing enterprise PaaS cloud, are good at automating test script generation, making the testing process more efficient. Other proposed framework [34] that uses layered symbolic execution (LSE) algorithm to automate testing for PaaS applications.

Software as a Service.

When testing SaaS pairwise testing, a form of black box testing, seems to be one of the best techniques [36, 58] to reduce the number of test cases when testing SaaS. Others [27, 28, 32] proposed framework to classify security alerts in SaaS cloud, automatically generate test cases that are based on Service Oriented Architecture, to test the scalability of SaaS applications [27]. Based on the MVC design pattern, [57] studied the construction and robustness testing of SaaS cloud computing data center.

Testing as a Service.

According to [37] the testing of mobile applications is a complex task due to the rapidly increasing number of different devices, operating systems, runtime environments, and network providers. TaaS can effectively address these difficulties. Prathibhan et al. [21] proposed a mobile application testing tool that serves as an interface between the user’s device and the cloud. Their tool systematically goes through a set of device emulators, where for each emulator, the execution of Android application is recorded as UI events, which are used to generate test scenarios. The proposed framework [37] consisted of multiple components: a cloud controller for administrative management, a hypervisor for virtual machine coordination, a mobile emulator manager for running different Android versions, a test automation manager for running GUI-based test cases was also implemented successfully.

In another study, the authors described a cloud-based framework [30] for conducting security tests on a set of mobile applications on the Android 4.0 OS. This work was further supported by [38] by the development of a framework for setting up large-scale emulator based mobile testing on the OpenStack cloud platform. The authors’ framework is composed of a request loader for generating mobile service requests, a resource provisioning engine for running load balancing algorithms, a graphical service monitor for displaying resource utilization measurements and request processing time.

2.3 Evaluation Methods

Case Studies.

Many authors have evaluated their methodologies by deploying their methodologies in real-world cloud systems or using real-world datasets as input. The authors selected Sunflow [19], an image rendering application, as the system under test (SUT) and FOSS-Cloud, an open source IT environment, as the runtime cloud provider. Others [17] selected OrangeHRM, an open source SaaS application as the SUT, and Amazon EC2 as the cloud infrastructure on top of which their TaaS system was built. Furthermore, the authors of [22] selected Olio, an open source social network application, as the SUT, and two public IaaS cloud providers, Amazon EC2 and Rackspace. Selection of OpenStack [13, 37] as the runtime cloud provider was an interesting approach. Mean response time was computed to detect related SLA violations and to determine the overhead of their proposed test system, the runtime cloud environment, and five mobile apps from Google Play Store. By inputting two datasets of security alerts generated from a production SaaS Cloud into their framework, the authors [28] evaluated the effectiveness of the framework for classifying alerts in the cloud.

The authors [30] describe selection of 100 mobile applications with varying vulnerabilities as input to their testing system and determined for their testing system correctly outputs the number of vulnerable applications, this work was further explored by [23] where, the authors selected a university-based social network application as the system under test and Azure Cloud as the runtime cloud provider. In a similar way others [26] selected Amazon EC2 as the runtime cloud provider and Yahoo Cloud Serving Benchmark, connected to MongoDB, as the system under test [33] selected a set of runtime providers, including Emulab, Amazon EC2, Open Cirrus, Wipro, and Elba, and a set of system of test, including RUBBoS, RUBiS, and Cloudstone. By deploying their framework on different cloud providers and for different systems, the authors [27] showed the richness of their solution to automated testing. Others [31] implemented their security testing system on the Microsoft SQL Server and used it to detect defects in 456 selected web applications.

Prototype Development.

A prototype system developed by [32] has the typical features of online banking was used to show how well their testing framework adapted to the prototype system. Others [34] evaluated their work based on how quickly test cases can be generated on this prototype system by implementing a prototype system on a virtual machine that includes all necessary PaaS components, such as language interpreter and operating system. They

Formal Proof.

To demonstrate the soundness of their proof, the authors used a formal proof [29] where, they proved that attacking their protocol through signature forgery has the same complexity as solving the RSA problem with large exponents.

Survey.

Conducting a survey of an experienced QA tester team showed to be effective to determine the effectiveness of the authors [22] proposed CLTF framework. The tester respondents wrote down the time they need to test fifty cloud services separately using CLTF and using JMeter, a popular open source testing tool. Based on these responses, the authors proved that their framework is better than JMeter in terms of time cost.

Other Studies.

Further studies were conducted in cloud computing including the introduction of a framework [56] to evaluate the risk involved in selecting a service provider, where the authors proposed a framework to evaluate the risk for various service providers in term of customers’ priorities. This framework helps customers of the cloud services to choose the providers with the best Service Level Agreement.

3 Discussion and Results

The research papers reviewed in this paper were obtained from electronic databases and were selected based on the title, abstract, and conclusion. In general, there is an uneven distribution of research effort within the area of cloud testing. This is demonstrated in Table 1, where it shows distribution according to the type of cloud services and testing.

From Fig. 1, it is clear that the majority of authors have demonstrated the usability of their proposals through real-world case studies indicating that real-world case study is the most convincing type of evaluation method in the research community.

4 Conclusion

As the field of cloud computing evolves, more research is needed to address the new challenges in the testing of cloud application and testing as a cloud service. This study contributes to a better understanding of the current trend of cloud testing research. Especially, testing of Platform-as-a-Service is a subarea that needs more investigations. Right now, there is no standard toolset for cloud testing. For future work, it may be worthwhile to investigate the different aspects of various commercialized testing tools through a comparative study. Nachiyappan et al. [39] have already compared five publicly available cloud testing products in terms of their pros, cons, and pricing model. There are more available products that can be incorporated into a comparative study and more aspects that need to be considered, including the scripting environment and API support for each testing product.