Zahra Sahaf
Vahid Garousi
Dietmar Pfahl
ABSTRACT
Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, since test automation usually requires substantial upfront investments, automation is not always more cost-effective than manual testing. To support decision-makers in finding the optimal degree of test automation in a given project, we propose in this paper a simulation model using the System Dynamics (SD) modeling technique. With the help of the simulation model, we can evaluate the performance of test processes with varying degrees of automation of test activities and help testers choose the most optimal cases. As the case study, we describe how we used our simulation model in the context of an Action Research (AR) study conducted in collaboration with a software company in Calgary, Canada. The goal of the study was to investigate how the simulation model can help decision-makers decide whether and to what degree the company should automate their test processes. As a first step, we compared the performances of the current fully manual testing with several cases of partly automated testing as anticipated for implementation in the partner company. The development of the simulation model as well as the analysis of simulation results helped the partner company to get a deeper understanding of the strengths and weaknesses of their current test process and supported decision-makers in the cost effective planning of improvements of selected test activities.
- G. Tassey, The economic impacts of inadequate infrastructure for software testing, RTI Final Report, National Institute of Standards and Technology, May 2002.Google Scholar
- E. Dustin, T. Garrett, and B. Gauf, Implementing automated software testing: How to save time and lower costs while raising quality: Addison-Wesley Professional, 2009. Google ScholarDigital Library
- K. Stobie, "Too much automation or not enough? When to automate testing," presented at the Pacific Northwest Software Quality Conference, 2009.Google Scholar
- D. J. Mosley and B. Posey, Just enough software test automation: Prentice Hall Professional, 2002. Google ScholarDigital Library
- D. Hoffman, "Cost Benefit Analysis of Test Automation", presented at Software Testing Analysis and Review (STAR West), 14 p, 1999.Google Scholar
- R. Ramler, S. Biffl, and P. Grünbacher, "Value-based management of software testing," in Value-Based Software Engineering, ed: Springer, 2006, pp. 225-244.Google Scholar
- R. Ramler and K. Wolfmaier, "Economic perspectives in test automation: balancing automated and manual testing with opportunity cost," in International workshop on Automation of Software Test, 2006, pp. 85-91. Google ScholarDigital Library
- L. Huang and B. Boehm, "How much software quality investment is enough: A value-based approach," IEEE Software, vol. 23, pp. 88-95, 2006. Google ScholarDigital Library
- Q. Li, M. Li, Y. Yang, Q. Wang, T. Tan, B. Boehm, et al., "Bridge the gap between software test process and business value: a case study," in Trustworthy Software Development Processes, ed: Springer, 2009, pp. 212-223. Google ScholarDigital Library
- J. W. Forrester, Industrial dynamics vol. 2: MIT press Cambridge, MA, 1961.Google Scholar
- D. M. Rafi, K. R. K. Moses, K. Petersen, and M. Mantyla, "Benefits and limitations of automated software testing: Systematic literature review and practitioner survey," in International Workshop on Automation of Software Test, 2012, pp. 36-42.Google Scholar
- O. Taipale, J. Kasurinen, K. Karhu, and K. Smolander, "Trade-off between automated and manual software testing," International Journal of System Assurance Engineering and Management, vol. 2, pp. 114-125, 2011.Google ScholarCross Ref
- M. I. Kellner and G. A. Hansen, "Software process modeling: a case study," in Proceedings of Annual Hawaii International Conference on System Sciences, 1989, pp. 175-188.Google Scholar
- T. K. Abdel-Hamid and S. E. Madnick, Software project dynamics: an integrated approach vol. 1: Prentice Hall Englewood Cliffs, 1991. Google ScholarDigital Library
- H. Zhang, B. Kitchenham, and D. Pfahl, "Software process simulation modeling: an extended systematic review," in New Modeling Concepts for Today’s Software Processes, ed: Springer, 2010, pp. 309-320. Google ScholarDigital Library
- J. S. Collofello, Z. Yang, J. D. Tvedt, D. Merrill, and I. Rus, "Modeling software testing processes," in Proceedings of Annual International Phoenix Conference on Computers and Communications, 1996, pp. 289-293.Google Scholar
- I. Rus, J. Collofello, and P. Lakey, "Software process simulation for reliability management," Journal of Systems and Software, vol. 46, pp. 173-182, 1999.Google ScholarCross Ref
- D. M. Raffo and W. Wakeland, "Assessing IV & V benefits using simulation,"" in Proceedings of NASA Goddard Software Engineering Workshop, 2003, pp. 97-101.Google Scholar
- V. Garousi, K. Khosrovian, and D. Pfahl, "A customizable pattern ‐based software process simulation model: design, calibration and application," Software Process: Improvement and Practice, vol. 14, pp. 165-180, 2009. Google ScholarDigital Library
- F. Glaiel, A. Moulton, and S. Madnick, "Agile Project Dynamics: A System Dynamics Investigation of Agile Software Development Methods," presented at the 31st International Conference of the System Dynamics Society 2013.Google Scholar
- L. Cao, B. Ramesh, and T. Abdel-Hamid, "Modeling dynamics in Agile software development," ACM Transactions on Management Information Systems, vol. 1, p. 5, 2010. Google ScholarDigital Library
- P. Ammann and J. Offutt, Introduction to software testing: Cambridge University Press, 2008. Google ScholarDigital Library
- R. Binder, Testing Object-oriented Software Testing: Models, Patterns, and Tools: Addison-Wesley Professional, 2000. Google ScholarDigital Library
- A. P. Mathur, Foundations of software testing: China Machine Press, 2008. Google ScholarDigital Library
- Hexawise Team. Hexawise – Pairwise Testing Made Easy. Available: http://www.hexawise.com, Last accessed: Dec. 2012Google Scholar
- E. Aranha and P. Borba, "An estimation model for test execution effort," in International Symposium on Empirical Software Engineering and Measurement, 2007, pp. 107- 116. Google ScholarDigital Library
- D. Pfahl, "ProSim/RA - Software Process Simulation in Support of Risk Assessment," in Value-based Software Engineering, S. Biffl, A. Aurum, B. Boehm, H. Erdogmus, and P. Grünbacher, Eds., ed: Springer Press, 2005, pp. 263- 286.Google Scholar
- C. Robson, Real world research: A resource for social scientists and practitioner-researchers vol. 2: Blackwell Oxford, 2011, 2d Edition.Google Scholar
Index Terms
- When to automate software testing? decision support based on system dynamics: an industrial case study
Recommendations
Economic perspectives in test automation: balancing automated and manual testing with opportunity cost
AST '06: Proceedings of the 2006 international workshop on Automation of software testTesting is a major cost factor in software development. Test automation has been proposed as one solution to reduce these costs. Test automation tools promise to increase the number of tests they run and the frequency at which they run them. So why not ...
A test generation solution to automate software testing
AST '08: Proceedings of the 3rd international workshop on Automation of software testThis paper describes the LEIRIOS Smart Testing™ approach to the functional validation of a subpart of the StarUML case study. This model-based testing process for test automation is shown from start to end: from functional requirements to the ...
When to automate software testing? A decision-support approach based on process simulation
Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, because test automation ...
Comments