Skip to main content
Springer Nature Link
Log in

Adoption and use of cloud-based testing in practice

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

This qualitative study addresses the adoption, utilization and effects of cloud-based testing in different organizational contexts. We approached the research problem by conducting thirty-five interviews with professionals in 20 organizations and applied grounded theory as the research method. The results indicate that cloud-based testing provides viable solutions to meet the testing needs within organizations. Cloud-based resources can be applied in performing various testing activities such as performance and multiplatform testing as well supporting practitioners in involving users during iterative development and testing. Cloud-based testing also adds value to practitioners by enabling easier management of the cloud-based testing resources and helping to produce improved end products. We use the results of the study to propose a strategy that can be used to assist practitioners in their decision-making processes towards adoption of cloud-based testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Armbrust, B., Fox, A., Griffith, R., Joseph, A., Katz, R., Konwinski, A., et al. (2010). A view of cloud computing. Communications of the ACM, 53(4), 50–58.

    Article  Google Scholar 

  • Buyya, R., Yeo, C. S., Venugopal, S., Broberg, J., & Brandic, I. (2009). Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future Generation Computer Systems, 25(6), 599–616.

    Article  Google Scholar 

  • Candea, G., Bucur, S., Zamfir, C. (2010). Automated software testing as a service. SoCC ‘10 proceedings of the 1st ACM symposium on cloud computing, pp. 155–160.

  • Ciortea, L., Zamfir, C., Bucur, S., Chipounov, V., & Candea, G. (2010). Cloud9: A software testing service. ACM SIGOPS Operating Systems Review, 43(4), 5–10.

    Article  Google Scholar 

  • Coleman, G., & O’Connor, R. (2007). Using grounded theory to understand software process improvement: A study of Irish software product companies. Information and Software Technology, 49(6), 654–667.

    Article  Google Scholar 

  • Creswell, D. L., & Miller, J. W. (2000). Determining validity in qualitative inquiry. Theory into Practice, 39(3), 124–130.

    Article  Google Scholar 

  • Gaisbauer, S., Kirschnick, J., Edwards. N., Rolia J. (2008). VATS: Virtualized-aware automated test service. Fifth international conference on quantitative evaluation of systems, pp. 93–102.

  • Ganon, Z., & Zilbershtein, I. E. (2009). Cloud-based performance testing of network management systems. Computer aided modeling and design of communication links and networks, pp. 1–6.

  • Gao, J., Bai, X., Tsai, W-T., Uehara, T. (2013). Testing as a Service (TaaS) on Clouds. IEEE 7th international symposium on service oriented system engineering (SOSE), pp. 212–223.

  • Gartner, (2009). Business models in global software ecosystem. Finnish Funding Agency for Technology and Innovation: Executive summary.

    Google Scholar 

  • Gasson, S. (2004). Rigor in grounded theory research: An interpretive perspective on generating theory from qualitative field studies. In the handbook of information systems research. pp. 79–102, IGI Global.

  • Hanawa, T., Banzai, T., Koizumi, H., Kanbayashi, R., Imada, T., Sato, M. (2010). Large-scale software testing environment using cloud computing technology for dependable parallel and distributed systems. Third international conference on software testing verification and validation workshops, pp. 428–433.

  • Incki, K., Ari, I., Sözer, H., (2012). A survey of software testing in the cloud, IEEE 6th international conference on software security and reliability companion (SERE-C), pp. 18–23.

  • ISO/IEC (2012). 29119-4 Systems and software engineering: Software testing—Part 4: Test techniques.

  • Jenkins, W., Vilkomir, S., Sharma, P., Pirocanac, G. (2011). Framework for testing cloud platforms and infrastructures. International conference on cloud and service computing, pp. 134–140.

  • Jensen, M., Schwenk, J., Gruschka, N., Iacono, L. L. (2009). On technical security issues in cloud computing. IEEE international conference on cloud computing, pp. 109–116.

  • Jun, W., & Meng, F. (2011). Software testing based on cloud computing. International conference on internet computing and information services, pp. 176–178.

  • Khajeh-Hosseini, A., Greenwood, D., Smith, J. W., & Sommerville, I. (2012). The cloud adoption toolkit: Supporting cloud adoption decisions in the enterprise. Software: Practice and Experience, 42, 447–465.

    Google Scholar 

  • Koong, C-S., Shih, C-H., Wu, C-C., Hsiung, P-A. (2013). The architecture of parallelized cloud-based automatic testing system, Seventh international conference on complex, intelligent, and software intensive systems (CISIS), pp. 467–470.

  • Lee, J. Y., Lee, J. W., Cheun, D. W., Kim, S. D. (2009). A quality model for evaluating software-as-a-service in cloud computing. Seventh ACIS international conference on software engineering research management and applications, pp. 261–266.

  • Liu, H., & Orban, D. (2009). Remote network labs: An on-demand network cloud for configuration testing. First ACM workshop on research on enterprise networking review, pp. 93–102.

  • Marston, S., Li, Z., Bandyopadhyay, S., Zhang, J., & Ghalsasi, A. (2011). Cloud computing — The business perspective. Decision Support Systems, 51(1), 176–189.

    Article  Google Scholar 

  • Maxwell, J. A. (2005). Qualitative research design: An interactive approach. Sage publications: London.

    Google Scholar 

  • Mell, P., & Grance, T. (2011) The NIST Definition of cloud computing. NIST Special Publication, http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf. Accessed 10 May 2012.

  • Mohammad, A. F., & Mcheick, H. (2011). Cloud services testing: An understanding. Procedia Computer Science, 5, 513–520. doi:10.1016/j.procs.2011.07.066.

    Article  Google Scholar 

  • Murthy, N. M. S., Suma, V. (2014). A study on cloud computing testing tools. The 48th annual convention of computer society of india - vol i. advances in intelligent systems and computing, pp. 605–612.

  • Nordic council of ministers. (2012). Nordic public sector cloud computing – a discussion paper. http://www.norden.org/en/publications/publikationer/2011-566. Accessed 12 October 2012.

  • Ojala, A., & Tyrväinen, P. (2011). Developing cloud business models: A case study on cloud gaming. IEEE Software, 28(4), 42–47.

    Article  Google Scholar 

  • Oriol, M., & Ullah, F. (2010). YETI on the cloud. Third international conference on software testing verification and validation workshops, pp. 434–437.

  • Parveen, T., Tilley, S. (2010). When to migrate software testing to the cloud?. Third international conference on software testing verification and validation workshops, pp. 424–427.

  • Priyanka, Chana, I., & Rana, A. (2012). Empirical evaluation of cloud-based testing techniques: A systematic review. ACM SIGSOFT Software Engineering Notes, 37(3), 1–9.

    Article  Google Scholar 

  • Riungu-Kalliosaari, L., Taipale, O., Smolander K. (2012a). Software testing as a service: Perceptions from practice. In S. Tilley & T. Parveen (Eds.), Testing as a service: Perspectives on an emerging discipline (pp. 196–215). IGI Global.

  • Riungu-Kalliosaari, L., Taipale, O., & Smolander, K. (2012). Testing in the cloud: Exploring the practice. IEEE Software, 29(2), 46–51.

    Article  Google Scholar 

  • Robinson, P., & Ragusa, C. (2011). Taxonomy and requirements rationalization for infrastructure in cloud-based software testing. IEEE third international conference on cloud computing technology and science, pp. 454–461.

  • Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2), 131–164.

    Article  Google Scholar 

  • Seaman, C. (1999). Qualitative methods in empirical studies of software engineering. IEEE Transactions on Software Engineering, 25(4), 557–572.

    Article  Google Scholar 

  • Strauss, A., & Corbin, J. (1990). basics of qualitative research: Grounded theory procedures and techniques.. London: Sage Publications.

    Google Scholar 

  • Sultan, N. A. (2011). Reaching for the ‘cloud’: How SMEs can manage. International Journal of Information Management, 31(3), 272–278.

    Article  Google Scholar 

  • Riungu L. M., Taipale, O., Smolander, K. (2010). Research issues for software testing in the cloud, IEEE second international conference on cloud computing technology and science, pp. 557–564.

  • Timilsina, U., Riungu-Kalliosaari, L., Taipale, O., Smolander, K., & Weijun, W. (2012). Security issues on testing of public cloud applications. New Technology of Library and Information Service, 28(11), 22–33.

    Google Scholar 

  • uTest, Microsoft security essentials, http://www.utest.com/spotlight/microsoft-security-essentials Accessed 10 May 2012.

  • uTest, Overview, http://www.utest.com/overview. Accessed 10 May 2012.

  • uTest, Testing types, http://www.utest.com/testing-types. Accessed 10 May 2012.

  • Wilson, P. (2011). Positive perspectives on cloud security. Information Security Technical Report, 16, 97–101. doi:10.1016/j.istr.2011.08.002.

    Article  Google Scholar 

  • Wu, J., Wang, C., Liu, Y., Zhang, L. (2011). AGARIC: A hybrid cloud based testing platform. International conference on cloud and service computing, pp. 87–94.

  • Yu, L., Tsai, W. T., Chen, X., Liu, L., Zhao, Y., Tang, L., Zhao, W. (2010). Testing as a service over cloud, Fifth IEEE international symposium on service oriented system engineering, pp. 181–188.

  • Zardari, S., & Bahsoon, R. (2011). Cloud adoption: A goal-oriented requirements engineering approach. 2nd international workshop on software engineering for cloud computing, pp. 29–35.

  • Zissis, D., & Lekkas, D. (2010). Addressing cloud computing security issues. Future Generation Computer Systems, 28(3), 583–592.

    Article  Google Scholar 

Download references

Acknowledgments

This study was supported by the ESPA project (http://www.soberit.hut.fi/espa/) and STX project (http://www2.it.lut.fi/project/STX/); both are funded by the Finnish Funding Agency for Technology and Innovations (TEKES) and by the companies involved in the projects. It was also supported, in part, by the Graduate School on Software Systems and Engineering (SoSE) and Lero - the Irish Software Engineering Research Centre (http://www.lero.ie) grant 10/CE/I1855.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Helsinki, Gustaf Hällströmin katu 2b, P.O. Box 68, 00014, Helsinki, Finland

    Leah Riungu-Kalliosaari

  2. Lero - The Irish Software Engineering Research Centre, University of Limerick, Limerick, Ireland

    Ita Richardson

  3. Department of Software Engineering and Information Management, Lappeenranta University of Technology, P.O. Box 20, 53851, Lappeenranta, Finland

    Ossi Taipale & Kari Smolander

Authors
  1. Leah Riungu-Kalliosaari
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ossi Taipale
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Kari Smolander
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Ita Richardson
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Leah Riungu-Kalliosaari.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Riungu-Kalliosaari, L., Taipale, O., Smolander, K. et al. Adoption and use of cloud-based testing in practice. Software Qual J 24, 337–364 (2016). https://doi.org/10.1007/s11219-014-9256-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-014-9256-0

Keywords