Skip to main content
SpringerLink
Log in

A dynamic random testing strategy in the context of cloud computing

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Dynamic random testing (DRT) strategy uses the testing results collected online to guide the selection of test cases, which can improve the fault detection effectiveness over random testing (RT) and random partition testing (RPT). The efficiency of current DRT is mainly concerned with traditional testing environment, i.e., the test cases are executed sequentially. The emergence of cloud testing provides the parallel testing environment to execute concurrent test cases simultaneously. If DRT strategy can be organically integrated into cloud environment, then the testing efficiency can be improved and the advantage of DRT can be maintained. However, how to select test cases in a parallel environment is a problem worth of exploration, and the resource allocation process should also be considered. Therefore, in this paper, we propose a dynamic random testing strategy in the context of cloud computing (DRT-C), which inherits the principle of DRT and adapts it into the cloud environment. In DRT-C, the test cases are selected and executed in parallel, and the testing profile is adjusted on the basis of concurrent testing results. Meanwhile, the testing cost, including time and resource costs, is optimized by a test case allocation method. The proposed strategy is evaluated by using 6 real-world software programs and 3 simulated software programs. The experimental results demonstrate that DRT-C significantly outperforms round robin schedule (RRS) and RPT-FIFO (first in first out) in terms of fault detection effectiveness, and is slightly better than DRT-FIFO. Besides, DRT-C outperforms other testing strategies in terms of testing costs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

All data included in this study are available upon reasonable request by contact with the corresponding author.

References

  • Arcuri, A., & Briand, L. (2011). Formal analysis of the probability of interaction fault detection using random testing. IEEE Transactions on Software Engineering, 38(5), 1088–1099. https://doi.org/10.1109/tse.2011.85

    Article  Google Scholar 

  • Arcuri, A., Iqbal, M. Z., & Briand, L. (2011). Random testing: Theoretical results and practical implications. IEEE Transactions on Software Engineering, 38(2), 258–277.

    Article  Google Scholar 

  • Arcuri, A., & Bri, L. (2014). A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing Verification and Reliability, 24(3), 219–250. https://doi.org/10.1002/stvr.1486

    Article  Google Scholar 

  • Bertolino, A., Miranda, B., Pietrantuono, R., & Russo, S. (2021). Adaptive test case allocation, selection and generation using coverage spectrum and operational profile. IEEE Transactions on Software Engineering, 47(5), 881–898. https://doi.org/10.1109/TSE.2019.2906187

    Article  Google Scholar 

  • Boland, P. J., Singh, H., & Cukic, B. (2003). Comparing partition and random testing via majorization and Schur functions. IEEE Transactions on Software Engineering, 29(1), 88–94. https://doi.org/10.1109/TSE.2003.1166591

    Article  Google Scholar 

  • Cai, K. Y. (2002). Optimal software testing and adaptive software testing in the context of software cybernetics. Information and Software Technology, 44(14), 841–855. https://doi.org/10.1016/s0950-5849(02)00108-8

    Article  Google Scholar 

  • Cai, K. Y., Jing, T., & Bai, C. G. (2005). Partition testing with dynamic partitioning. In 29th Annual International Computer Software and Applications Conference (COMPSAC'05) (Vol. 2, pp. 113-116). IEEE. https://doi.org/10.1109/COMPSAC.2005.118

  • Cai, K. Y., Gu, B., Hu, H., & Li, Y. C. (2007). Adaptive software testing with fixed-memory feedback. Journal of Systems and Software, 80(8), 1328–1348. https://doi.org/10.1016/j.jss.2006.11.008

    Article  Google Scholar 

  • Cai, K. Y., Hu, H., Jiang, C. H., & Ye, F. (2009). Random testing with dynamically updated test profile. In Proceedings of the 20th International Symposium On Software Reliability Engineering (ISSRE 2009) (pp. 1-2).

  • Cañizares, P. C., Núñez, A., de Lara, J., & Llana, L. (2020). MT-EA4Cloud: A methodology for testing and optimising energy-aware cloud systems. Journal of Systems and Software, 163, 110522.

  • Calheiros, R. N., Ranjan, R., Beloglazov, A., De Rose, C. A., & Buyya, R. (2011). CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and experience, 41(1), 23-50.

  • Chen, T. Y., & Yu, Y. T. (1994). On the relationship between partition and random testing. IEEE Transactions on Software Engineering, 20(12), 977–980. https://doi.org/10.1109/32.368132

    Article  Google Scholar 

  • Chen, T. Y., & Yu, Y. T. (1996). On the expected number of failures detected by subdomain testing and random testing. IEEE Transactions on Software Engineering, 22(2), 109–119. https://doi.org/10.1109/32.485221

    Article  Google Scholar 

  • Chen, T. Y., Tse, T. H., & Yu, Y. T. (2001). Proportional sampling strategy: A compendium and some insights. Journal of Systems and Software, 58(1), 65–81. https://doi.org/10.1016/S0164-1212(01)00028-0

    Article  Google Scholar 

  • Chen, T. Y., & Merkel, R. (2006). Efficient and effective random testing using the Voronoi diagram. In Australian Software Engineering Conference (ASWEC'06) (pp. 6-pp). IEEE.

  • Chen, T. Y., & Merkel, R. (2007). Quasi-random testing. IEEE Transactions on Reliability, 56(3), 562–568. https://doi.org/10.1109/TC.2015.2455981

    Article  Google Scholar 

  • Chen, J., Zhu, L., Chen, T. Y., Towey, D., Kuo, F. C., Huang, R., & Guo, Y. (2018). Test case prioritization for object-oriented software: An adaptive random sequence approach based on clustering. Journal of Systems and Software, 135, 107–125. https://doi.org/10.1016/j.jss.2017.09.031

    Article  Google Scholar 

  • Chen, M., Challita, U., Saad, W., Yin, C., & Debbah, M. (2019). Artificial neural networks-based machine learning for wireless networks: A tutorial. IEEE Communications Surveys & Tutorials, 21(4), 3039–3071. https://doi.org/10.1109/COMST.2019.2926625

    Article  Google Scholar 

  • Chan, F. T., Chen, T. Y., Mak, I. K., & Yu, Y. T. (1996). Proportional sampling strategy: Guidelines for software testing practitioners. Information and Software Technology, 38(12), 775–782. https://doi.org/10.1016/0950-5849(96)01103-2

    Article  Google Scholar 

  • Chang, V., Kuo, Y. H., & Ramachandran, M. (2016). Cloud computing adoption framework: A security framework for business clouds. Future Generation Computer Systems, 57, 24–41.

    Article  Google Scholar 

  • Chang, X., Xia, R., Muppala, J. K., Trivedi, K. S., & Liu, J. (2018). Effective modeling approach for IaaS data center performance analysis under heterogeneous workload. IEEE Transactions on Cloud Computing, 6(4), 991–1003. https://doi.org/10.1109/TCC.2016.2560158

    Article  Google Scholar 

  • Chow, C., Chen, T. Y., & Tse, T. H. (2013). The ART of divide and conquer: An innovative approach to improving the efficiency of adaptive random testing. In 2013 13th International Conference on Quality Software (pp. 268-275). IEEE. https://doi.org/10.1109/QSIC.2013.19

  • Gelenbe, E., & Fourneau, J. M. (2014). Random neural networks with multiple classes of signals. Neural Computation, 11(4), 953–963. https://doi.org/10.1162/089976699300016520

    Article  Google Scholar 

  • Gutjahr, W. J. (1999). Partition testing vs. random testing: The influence of uncertainty. IEEE Transactions on Software Engineering, 25(5), 661-674. https://doi.org/10.1109/32.815325

  • Hao, D., Zhang, L., Zhang, L., Rothermel, G., & Mei, H. (2014). A unified test case prioritization approach. ACM Transactions on Software Engineering and Methodology (TOSEM), 24(2), 1-31.

  • Hao, D., Zhang, L., Zang, L., Wang, Y., Wu, X., & Xie, T. (2016). To be optimal or not in test-case prioritization. IEEE Transactions on Software Engineering, 42(5), 490–505. https://doi.org/10.1109/TSE.2015.2496939

    Article  Google Scholar 

  • Hu, B., Cao, Z., & Zhou, M. (2022). Scheduling real-time parallel applications in cloud to minimize energy consumption. IEEE Transactions on Cloud Computing, 10(1), 662–674. https://doi.org/10.1109/TCC.2019.2956498

    Article  Google Scholar 

  • Huang, R., Liu, H., Xie, X., & Chen, J. (2015). Enhancing mirror adaptive random testing through dynamic partitioning. Information and Software Technology, 67, 13–29. https://doi.org/10.1016/j.infsof.2015.06.003

    Article  Google Scholar 

  • Hwang, K., Bai, X., Shi, Y., Li, M., Chen, W. G., & Wu, Y. (2015). Cloud performance modeling with benchmark evaluation of elastic scaling strategies. IEEE Transactions on Parallel and Distributed Systems, 27(1), 130–143.

    Article  Google Scholar 

  • Kazmi, R., Jawawi, D. N., Mohamad, R., & Ghani, I. (2017). Effective regression test case selection: A systematic literature review. ACM Computing Surveys (CSUR), 50(2), 29.

    Google Scholar 

  • Kuo, F. C. (2009). An indepth study of mirror adaptive random testing. In 2009 Ninth International Conference on Quality Software (pp. 51-58). IEEE.

  • Kumar, R., Jain, K., Maharwal, H., Jain, N., & Dadhich, A. (2014). Apache cloudstack: Open source infrastructure as a service cloud computing platform. Proceedings of the International Journal of advancement in Engineering technology, Management and Applied Science, 111, 116.

  • Li, Y., Yin, B. B., Lv, J., & Cai, K. Y. (2015). Approach for test profile optimization in dynamic random testing. In 2015 IEEE 39th Annual Computer Software and Applications Conference (Vol. 3, pp. 466-471). IEEE. https://doi.org/10.1109/COMPSAC.2015.257

  • Lv, J., Hu, H., & Cai, K. Y. (2011). A sufficient condition for parameters estimation in dynamic random testing. In 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops (pp. 19-24). IEEE. https://doi.org/10.1109/COMPSACW.2011.14

  • Lv, J., Hu, H., Cai, K. Y., & Chen, T. Y. (2014). Adaptive and random partition software testing. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 44(12), 1649–1664. https://doi.org/10.1109/TSMC.2014.2318019

    Article  Google Scholar 

  • Mei, H., Hao, D., Zhang, L., Zhang, L., Zhou, J., & Rothermel, G. (2012). A static approach to prioritizing JUnit test cases. IEEE Transactions on Software Engineering, 38(6), 1258–1275.

    Article  Google Scholar 

  • Ostrand, T. J., & Balcer, M. J. (1988). The category-partition method for specifying and generating functional tests. Communications of the ACM, 31(6), 676–686. https://doi.org/10.1145/62959.62964

    Article  Google Scholar 

  • Pandey, S., Wu, L., Guru, S. M., & Buyya, R. (2010). A particle swarm optimization-based heuristic for scheduling workflow applications in cloud computing environments. In 2010 24th IEEE International Conference on Advanced Information Networking and Applications (pp. 400-407). IEEE.

  • Pei, H., Yin, B., & Xie, M. (2018). Dynamic random testing strategy for test case optimization in cloud environment. IEEE international symposium on software reliability engineering workshops (ISSRE). Memphis, TN, USA, 2018, 148–149.

    Google Scholar 

  • Pei, H. Y., Cai, K. Y., Yin, B. B., Mathur, A. P., & Xie, M. (2019). Dynamic random testing: Technique and experimental evaluation. IEEE Transactions on Reliability, 68(3), 872–892. https://doi.org/10.1109/TR.2019.2911593

    Article  Google Scholar 

  • Pei, H. Y., Yin, B. B., Xie, M., & Cai, K. Y. (2021). Dynamic random testing with test case clustering and distance-based parameter adjustment. Information and Software Technology, 131(12), 106470. https://doi.org/10.1016/j.infsof.2020.106470

    Article  Google Scholar 

  • Pahl, C., Brogi, A., Soldani, J., & Jamshidi, P. (2019). Cloud container technologies: A state-of-the-art review. IEEE Transactions on Cloud Computing, 7(3), 677–692. https://doi.org/10.1109/TCC.2017.2702586

    Article  Google Scholar 

  • Rasmussen, R. V., & Trick, M. A. (2008). Round robin scheduling–A survey. European Journal of Operational Research, 188(3), 617–636. https://doi.org/10.1016/j.ejor.2007.05.046

    Article  MathSciNet  MATH  Google Scholar 

  • Song, C., Han, G., & Zeng, P. (2022). Cloud computing based demand response management using deep reinforcement learning. IEEE Transactions on Cloud Computing, 10(1), 72–81. https://doi.org/10.1109/TCC.2021.3117604

    Article  Google Scholar 

  • Sun, C., Dai, H., Liu, H., Chen, T. Y., & Cai, K. Y. (2019). Adaptive partition testing. IEEE Transactions on Computers, 68(2), 157–169. https://doi.org/10.1109/TC.2018.2866040

    Article  MathSciNet  MATH  Google Scholar 

  • Tao, Y., Qiu, J., & Lai, S. (2022). A hybrid cloud and edge control strategy for demand responses using deep reinforcement learning and transfer learning,". IEEE Transactions on Cloud Computing, 10(1), 56–71. https://doi.org/10.1109/TCC.2021.3117580

    Article  Google Scholar 

  • Wang, L., & Gelenbe, E. (2018). Adaptive dispatching of tasks in the cloud. IEEE Transactions on Cloud Computing, 6(1), 33–45. https://doi.org/10.1109/TCC.2015.2474406

    Article  Google Scholar 

  • Wang, X., Han, Y., Leung, V. C., Niyato, D., Yan, X., & Chen, X. (2020). Convergence of edge computing and deep learning: A comprehensive survey. IEEE Communications Surveys and Tutorials, 22(2), 869–904. https://doi.org/10.1109/COMST.2020.2970550

    Article  Google Scholar 

  • Xiao, P., Yin, Y., Liu, B., Jiang, B., & Malaiya, Y. K. (2017). Adaptive testing based on moment estimation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 50(3), 911–922. https://doi.org/10.1109/TSMC.2017.2761767

    Article  Google Scholar 

  • Yang, Z., Yin, B., Lv, J., Cai, K., Yau, S. S., & Yu, J. (2014). Dynamic random testing with parameter adjustment. In 2014 IEEE 38th International Computer Software and Applications Conference Workshops (pp. 37-42). IEEE. https://doi.org/10.1109/COMPSACW.2014.10

  • Yang, H. J., Feng, C., & Suleiman, A. (2017). Modern software cybernetics: New trends. Journal of Systems and Software, 124, 169–186. https://doi.org/10.1016/j.jss.2016.08.095

    Article  Google Scholar 

  • Yao, J., Maleki Shoja, B., & Tabrizi, N. (2019). An Overview of Cloud Computing Testing Research. In International Conference on Cloud Computing (pp. 303-313). Springer, Cham.

  • Zhang, L., Yin, B. B., Lv, J., Cai, K. Y., Yau, S. S., & Yu, J. (2014a). A history-based dynamic random software testing. In 2014 IEEE 38th International Computer Software and Applications Conference Workshops (pp. 31-36). IEEE. https://doi.org/10.1109/COMPSACW.2014a.9

  • Zhang, Y. P., Qin, J., & Zhai, Z. (2014b). The research and Improvement of MapReduce cluster scheduling strategy based on ant colony optimization. Applied Mechanics and Materials, 513, 1989–1993. https://doi.org/10.4028/www.scientific.net/AMM.513-517.1989

    Article  Google Scholar 

Download references

Funding

This work is supported in part by National Key R&D Program of China under Grant 2021YFB1600601, in part by the National Natural Science Foundation of China under Grant 61772055 and Grant 61872169.

Author information

Authors and Affiliations

  1. Department of Automatic Control, School of Automation Science and Electrical Engineering, Beihang University, Beijing, People’s Republic of China

    Hanyu Pei, Beibei Yin, Linzhi Huang & Kai-Yuan Cai

Authors
  1. Hanyu Pei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beibei Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linzhi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai-Yuan Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Beibei Yin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pei, H., Yin, B., Huang, L. et al. A dynamic random testing strategy in the context of cloud computing. Software Qual J 31, 243–277 (2023). https://doi.org/10.1007/s11219-022-09599-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-022-09599-w

Keywords

Search

Navigation