skip to main content
10.1145/2993288.2993302acmotherconferencesArticle/Chapter ViewAbstractPublication PagessastConference Proceedingsconference-collections
research-article

Relevance and Impact of Critical Factors of Success in Software Test Automation lifecycle: A Survey

Published: 19 September 2016 Publication History

Abstract

Context:Software has increased in size and complexity. It has increased the amount of time and money required for testing. Many organizations have invested in software test automation (STA) waiting for reducing costs and improving the testing process. However, less than 50% of them reach the expected objectives related to software test automation mainly due to the lack of a clear understanding of what is involved. Objective: There is a documented gap between the academic and practitioners' points of views about software test automation. This paper has two main objectives. First, evaluating the relevance of 12 critical factors of success (CFS) in software test automation collected from the technical literature according to researchers views. Second, evaluating the impact of each of them on a basic software test automation lifecycle (BSTAL). Method: To archive the above objectives, we have performed a survey with software test practitioners. Each participant was invited by e-mail to answer an electronic survey to evaluate the relevance of CFSs. A cutoff value was defined to classify factors according to their relevant levels. Factors with relevant levels under 50% are not considered relevant. Relevant level was calculated in two ways: based on the entire population and based on subsets of participants grouped by their years of working in software testing. Results: We observed differences between academic and practitioner evaluations about the relevance level as well between the evaluations given by participants with different level of experience in software testing. We also observed that each factor impacts the BSTAL phases differently and they can be more or less relevant according to the phase under observation.

References

[1]
J. Andersin. Tpi-a model for test process improvement. In Seminar. University of Helsinki, Helsinki-Finland. Citeseer, 2004.
[2]
Atech. Atech, 2015. "http://www.atech.com.br".
[3]
W. Belassi and O. I. Tukel. A new framework for determining critical success/failure factors in projects. International journal of project management, 14(3):141--151, 1996.
[4]
A. Bertolino. An overview of automated software testing. Journal of Systems and Software, 15(2):133--138, 1991.
[5]
BSTQB. Brazilian software testing qualifications board - bstqb, 2015. "http://www.bstqb.org.br/".
[6]
A. Cervantes. Exploring the use of a test automation framework. In 2009 IEEE Aerospace conference, pages 1--9. IEEE, 2009.
[7]
E. F. Collins et al. Software test automation practices in agile development environment: An industry experience report. In Proceedings of the 7th International Workshop on Automation of Software Test, pages 57--63. IEEE Press, 2012.
[8]
A. C. Dias-Neto and G. H. Travassos. Evaluation of model-based testing techniques selection approaches: An external replication. In Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement., pages 269--278. IEEE Computer Society, 2009.
[9]
E. Dustin, J. Rashka, and J. Paul. Automated software testing: introduction, management, and performance. Addison-Wesley Professional, 1999.
[10]
D. Graham and M. Fewster. Experiences of test automation: case studies of software test automation. Addison-Wesley Professional, 2012.
[11]
M. Hamberg. Basic statistics: A modern approach. Journal of the Royal Statistical Society, 143(1), 1980.
[12]
D. Hoffman. Cost benefits analysis of test automation. STAR West, 99, 1999.
[13]
INDT. Indt - instituto de desenvolvimento tecnológico, 2016. "www.indt.org.br/".
[14]
K. Karhu, T. Repo, O. Taipale, and K. Smolander. Empirical observations on software testing automation. Proceedings - 2nd International Conference on Software Testing, Verification, and Validation, ICST 2009, pages 201--209, 2009.
[15]
J. Kasurinen, O. Taipale, and K. Smolander. Software test automation in practice: empirical observations. Advances in Software Engineering, 2010, 2010.
[16]
H. Kerzner. Advanced project management: Best practices on implementation. John Wiley & Sons, 2004.
[17]
J. Lee, S.-Y. Kang, and D. Lee. Survey on software testing practices. Software, IET, 6(3):275--282, 2012.
[18]
B. Pettichord. Seven Steps to Test Automation Success. Star, November 2001.
[19]
P. Pocatilu. Automated software testing process. Economy Informatics, 1:97--99, 2002.
[20]
P. Poon and C. Wagner. Critical success factors revisited: success and failure cases of information systems for senior executives. Decision support systems, 30(4):393--418, 2001.
[21]
PRODAM. Processamento de dados do amazonas -prodam, 2015. "www.prodam.am.gov.br/".
[22]
D. M. Rafi, K. R. K. Moses, K. Petersen, and M. V. Mäntylä. Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In Proceedings of the 7th International Workshop on Automation of Software Test, pages 36--42. IEEE Press, 2012.
[23]
R. Ramler and K. Wolfmaier. Economic perspectives in test automation: balancing automated and manual testing with opportunity cost. In Proceedings of the 2006 international workshop on Automation of software test, pages 85--91. ACM, 2006.
[24]
A. Rodrigues, A. Dias-Neto, and A. Bezerra. Tapn: Test automation's pyramid of needs. In Proceedings of the XIV Brazilian Symposium on Software Quality, pages 122--136. Brazilian Computation Society, 2015.
[25]
Z. Sahaf, V. Garousi, D. Pfahl, R. Irving, and Y. Amannejad. When to automate software testing? decision support based on system dynamics: an industrial case study. In Proceedings of the 2014 International Conference on Software and System Process, pages 149--158. ACM, 2014.
[26]
C. Schwaber and M. Gilpin. Evaluating automated functional testing tools. Forrester Research, 2005.
[27]
K. Schwaber. Scrum development process. In Business Object Design and Implementation, pages 117--134. Springer, 1997.
[28]
SIDIA. Samsung instituto de desenvolvimento informática da amazônia (sidia), 2015. Av. Min. João G. de Souza, 880 - Distrito Industrial I, Manaus - AM, 69075--830.
[29]
O. Taipale, J. Kasurinen, K. Karhu, and K. Smolander. Trade-off between automated and manual software testing. International Journal of System Assurance Engineering and Management, 2(2):114--125, 2011.
[30]
F. Tech. "fundação paulo feitosa de tecnologia - fpf tech", 2015. "http://www.fpftech.com/".
[31]
D. Testes. Grupo de amigos profissionais em teste de software do distrito federal, 2015. "http://br.groups.yahoo.com/group/DFTestes/".
[32]
S. Thummalapenta, S. Sinha, N. Singhania, and S. Chandra. Automating test automation. In 2012 34th International Conference on Software Engineering (ICSE), pages 881--891. IEEE, 2012.
[33]
UFAM. Escola avançada em teste de software - eats, Aug 2014. Event held by the Federal University of Amazonas - UFAM in the period from Augost 13th to 15th of 2014.
[34]
E. v. Veenendaal and B. Wells. Test Maturity Model integration TMMi. Uitgeverij Tutein Nolthenius, 2012.
[35]
K. Wiklund, D. Sundmark, S. Eldh, and K. Lundvist. Impediments for automated testing - An empirical analysis of a user support discussion board. Proceedings - IEEE 7th International Conference on Software Testing, Verification and Validation, ICST 2014, pages 113--122, 2014.
[36]
T. Wissink and C. Amaro. Successful test automation for software maintenance. In 2006 22nd IEEE International Conference on Software Maintenance, pages 265--266. IEEE, 2006.

Cited By

View all
  • (2024)SoK: Automated Software Testing for TLS LibrariesProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670871(1-12)Online publication date: 30-Jul-2024
  • (2023)A conceptual model supporting decision-making for test automation in Agile-based Software DevelopmentData & Knowledge Engineering10.1016/j.datak.2022.102111144:COnline publication date: 1-Mar-2023
  • (2023)Nature-inspired metaheuristic methods in software testingSoft Computing10.1007/s00500-023-08382-828:2(1503-1544)Online publication date: 8-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SAST '16: Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing
September 2016
154 pages
ISBN:9781450347662
DOI:10.1145/2993288
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

  • SBC: Sociedade Brasileira de Computação

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. critical factors of success
  2. lifecyle
  3. test automation

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SAST '16

Acceptance Rates

SAST '16 Paper Acceptance Rate 15 of 34 submissions, 44%;
Overall Acceptance Rate 45 of 92 submissions, 49%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)31
  • Downloads (Last 6 weeks)2
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)SoK: Automated Software Testing for TLS LibrariesProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670871(1-12)Online publication date: 30-Jul-2024
  • (2023)A conceptual model supporting decision-making for test automation in Agile-based Software DevelopmentData & Knowledge Engineering10.1016/j.datak.2022.102111144:COnline publication date: 1-Mar-2023
  • (2023)Nature-inspired metaheuristic methods in software testingSoft Computing10.1007/s00500-023-08382-828:2(1503-1544)Online publication date: 8-Jun-2023
  • (2022)Evidence and perceptions on GUI test automation - An Exploratory StudyProceedings of the 15th Innovations in Software Engineering Conference10.1145/3511430.3511442(1-10)Online publication date: 24-Feb-2022
  • (2022)Improving test automation maturity: A multivocal literature reviewSoftware Testing, Verification and Reliability10.1002/stvr.180432:3Online publication date: 15-Feb-2022
  • (2021)A survey on software test automation return on investment, in organizations predominantly from Bengaluru, IndiaInternational Journal of Engineering Business Management10.1177/1847979021106204413Online publication date: 17-Dec-2021
  • (2018)Maintaining behaviour driven development specifications: Challenges and opportunities2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER.2018.8330207(175-184)Online publication date: Mar-2018

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media