research-article

Testing Techniques Selection: A Systematic Mapping Study

Authors:

Silvana M. Melo,

Paulo Sergio Lopes de Souza,

Simone R. S. SouzaAuthors Info & Claims

SBES '19: Proceedings of the XXXIII Brazilian Symposium on Software Engineering

Pages 347 - 356

https://doi.org/10.1145/3350768.3352571

Published: 23 September 2019 Publication History

Abstract

[Context] Software projects must consider the selection of testing techniques and criteria during their life cycles. This practice increases the chances of testing activity to be appropriately performed. In a previous work, an infrastructure to support the selection of testing techniques was proposed for the context of concurrent software. This infrastructure considers information (attributes) of the project to make the selection closer to the testers need. [Objective] This paper extends the previous work by identifying new studies concerning testing techniques selection, project attributes that can be used for this selection and which approaches can be employed to support the combined selection. [Method] A mapping study was conducted and a total of 15 primary studies, published in the last 20 years were selected. Information about approaches to testing techniques selection was analyzed and classified. [Results] The following results were obtained: (i) existing approaches for selection of testing techniques; (ii) proposition of a taxonomy of selection approaches; (iii) characterization of attributes to offer the support for the selection of a testing technique; and (iv) identification of approaches that perform combined selection of testing techniques. [Conclusion] Combining testing techniques is essential to improve the testing activity quality by finding different failure categories and supplementing other techniques limitations. This paper describes an initiative to offer support for the construction of new or combined strategies for the selection of testing techniques capable of being used in practice.

References

[1]

Philip Achimugu, Ali Selamat, Roliana Ibrahim, and Mohd Naz'ri Mahrin. 2014. A systematic literature review of software requirements prioritization research. Information and software technology 56, 6 (2014), 568--585.

[2]

Norma Barrett, Simon Martin, and Chryssa Dislis. 1999. Test process optimization: Closing the gap in the defect spectrum. In Test Conference, 1999. Proceedings. International. IEEE, 124--129.

[3]

Domenico Cotroneo, Roberto Pietrantuono, and Stefano Russo. 2013. A learning-based method for combining testing techniques. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 142--151.

Digital Library

[4]

Domenico Cotroneo, Roberto Pietrantuono, and Stefano Russo. 2013. Testing techniques selection based on ODC fault types and software metrics. Journal of Systems and Software 86, 6 (2013), 1613--1637.

Digital Library

[5]

Aurélio da Silva Grande, Arilo Claudio Dias Neto, and Rosiane de Freitas Rodrigues. 2012. Providing trade-off techniques subsets to improve software testing effectiveness: using evolutionary algorithm to support software testing techniques selection by a web tool. In Advances in Artificial Intelligence-SBIA 2012. Springer, 222--232.

[6]

Diego Dermeval, Jéssyka Vilela, Ig Ibert Bittencourt, Jaelson Castro, Seiji Isotani, and Patrick Brito. 2014. A systematic review on the use of ontologies in requirements engineering. In Software Engineering (SBES), 2014 Brazilian Symposium on. IEEE, 1--10.

[7]

Arilo Claudio Dias-Neto and Guilherme Horta Travassos. 2009. Evaluation of {model-based} testing techniques selection approaches: An external replication. In Empirical Software Engineering and Measurement, 2009. ESEM 2009. 3rd International Symposium on. IEEE, 269--278.

Digital Library

[8]

Arilo Claudio Dias-Neto and Guilherme Horta Travassos. 2014. Supporting the combined selection of model-based testing techniques. IEEE Transactions on Software Engineering 40, 10 (2014), 1025--1041.

[9]

Wei Ding, Peng Liang, Antony Tang, and Hans Van Vliet. 2014. Knowledge-based approaches in software documentation: A systematic literature review. Information and Software Technology 56, 6 (2014), 545--567.

Digital Library

[10]

Tore Dybå and Torgeir Dingsøyr. 2008. Empirical studies of agile software development: A systematic review. Information and software technology 50, 9 (2008), 833--859.

[11]

Sheikh Umar Farooq and SMK Quadri. 2013. Empirical evaluation of software testing techniques-need, issues and mitigation. Software engineering: an international Journal 3, 3 (2013), 41--51.

[12]

Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1--26.

[13]

Sara Mahdavi-Hezavehi, Matthias Galster, and Paris Avgeriou. 2013. Variability in quality attributes of service-based software systems: A systematic literature review. Information and Software Technology 55, 2 (2013), 320--343.

Digital Library

[14]

Silvana Morita Melo. 2018. A Framework for systematic testing techniques evaluation applied to concurrent programming. Ph.D. Dissertation. Universidade de São Paulo.

[15]

Silvana Morita Melo, Simone Rocio Senger de Souza, Paulo Sergio Lopes de Souza, and Jeffrey C Carver. 2017. How to test your concurrent software: an approach for the selection of testing techniques. In Proceedings of the 4th ACM SIGPLAN International Workshop on Software Engineering for Parallel Systems. ACM, 42--43.

Digital Library

[16]

Felipe Moreira Moura, Silvana Morita Melo, Simone do Rocio Senger de Souza, et al. 2018. SeleCTT: ferramenta para automatização do processo de seleção de técnicas de teste de software concorrente. (2018).

[17]

Glenford J Myers, Corey Sandler, and Tom Badgett. 2011. The art of software testing. John Wiley & Sons.

Digital Library

[18]

Arilo Claudio Dias Neto, Rosiane de Freitas Rodrigues, and Guilherme Horta Travassos.2011. Porantim-opt: Optimizing the combined selection of model-based testing techniques. In Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on. IEEE, 174--183.

[19]

Arilo Claudio Dias Neto and Guilherme Horta Travassos. 2009. Porantim: An approach to support the combination and selection of Model-based Testing techniques. In Automation of Software Test, 2009. AST'09. ICSE Workshop on. IEEE, 1--9.

[20]

Gregg Rothermel and Mary Jean Harrold. 1996. Analyzing regression test selection techniques. IEEE Transactions on software engineering 22, 8 (1996), 529--551.

Digital Library

[21]

Silvia Santa Isabel and Guilherme H Travassos. 2011. Features of software testing techniques for use in projects web. 14th Ibero-American Conference on Software Engineering and 14th Workshop on Requirements Engineering, CIbSE 2011 (2011), 129--142.

[22]

Muhammad Usman, Emilia Mendes, Francila Weidt, and Ricardo Britto. 2014. Effort estimation in agile software development: a systematic literature review. In Proceedings of the 10th international conference on predictive models in software engineering. ACM, 82--91.

Digital Library

[23]

Sira Vegas. 2004. Identifying the relevant information for software testing technique selection. In Empirical Software Engineering, 2004. ISESE'04. Proceedings. 2004 International Symposium on. IEEE, 39--48.

[24]

Sira Vegas and Victor Basili. 2005. A characterisation schema for software testing techniques. Empirical Software Engineering 10, 4 (2005), 437--466.

Digital Library

[25]

Sira Vegas, Natalia Juristo, and Victor Basili. 2006. Packaging experiences for improving testing technique selection. Journal of Systems and Software 79, 11 (2006), 1606--1618.

[26]

Monisha Victor and Nitin Upadhyay. 2011. Selection of software testing technique: A multi criteria decision making approach. In Trends in Computer Science, Engineering and Information Technology. Springer, 453--462.

[27]

Margaret A Wojcicki and Paul Strooper. 2007. An iterative empirical strategy for the systematic selection of a combination of verification and validation technologies. In Proceedings of the 5th International Workshop on Software Quality. IEEE Computer Society, 9.

Digital Library

Cited By

Cerqueira LSilva JAlvim ÍMendonça MSantos J(2022)The who, what and how of the current research at the Brazilian Symposium on Software EngineeringProceedings of the XXXVI Brazilian Symposium on Software Engineering10.1145/3555228.3555241(11-20)Online publication date: 5-Oct-2022
https://dl.acm.org/doi/10.1145/3555228.3555241
Tran HUnterkalmsteiner MBörstler JAli N(2021)Assessing test artifact quality—A tertiary studyInformation and Software Technology10.1016/j.infsof.2021.106620139:COnline publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1016/j.infsof.2021.106620
Santos IMelo SLopes de Souza PSouza S(2020)Towards a unified catalog of attributes to guide industry in software testing technique selection2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00071(398-407)Online publication date: Oct-2020
https://doi.org/10.1109/ICSTW50294.2020.00071
Show More Cited By

Testing Techniques Selection: A Systematic Mapping Study
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Contributions to improve the combined selection of concurrent software testing techniques
SAST '20: Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing

[Context] There are a variety of testing techniques available that present different and often complementary characteristics (e.g., cost of application, effectiveness in revealing defects, types of defects). Considering these complementary ...
Efficient, effective regression testing using safe test selection techniques
Porantim-Opt: Optimizing the Combined Selection of Model-Based Testing Techniques
ICSTW '11: Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops

The combination of testing techniques is considered an effective strategy to evaluate a software product. However, the selection of which techniques to combine in a software project has been an interesting challenge in the Software Engineering field. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBES '19: Proceedings of the XXXIII Brazilian Symposium on Software Engineering

September 2019

583 pages

ISBN:9781450376518

DOI:10.1145/3350768

General Chairs:
Ivan Machado
UFBA
,
Rodrigo Souza
UFBA
,
Rita Suzana Maciel
UFBA
,
Claudio Sant'Anna
UFBA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SBC: Sociedade Brasileira de Computação

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Fundação de Amparo à Pesquisa do Estado de São Paulo

Conference

SBES 2019

SBES 2019: XXXIII Brazilian Symposium on Software Engineering

September 23 - 27, 2019

Salvador, Brazil

Acceptance Rates

SBES '19 Paper Acceptance Rate 67 of 153 submissions, 44%;

Overall Acceptance Rate 147 of 427 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
199
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cerqueira LSilva JAlvim ÍMendonça MSantos J(2022)The who, what and how of the current research at the Brazilian Symposium on Software EngineeringProceedings of the XXXVI Brazilian Symposium on Software Engineering10.1145/3555228.3555241(11-20)Online publication date: 5-Oct-2022
https://dl.acm.org/doi/10.1145/3555228.3555241
Tran HUnterkalmsteiner MBörstler JAli N(2021)Assessing test artifact quality—A tertiary studyInformation and Software Technology10.1016/j.infsof.2021.106620139:COnline publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1016/j.infsof.2021.106620
Santos IMelo SLopes de Souza PSouza S(2020)Towards a unified catalog of attributes to guide industry in software testing technique selection2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00071(398-407)Online publication date: Oct-2020
https://doi.org/10.1109/ICSTW50294.2020.00071
Santos IFilho JSouza S(2020)A survey on the practices of mobile application testing2020 XLVI Latin American Computing Conference (CLEI)10.1109/CLEI52000.2020.00034(232-241)Online publication date: Oct-2020
https://doi.org/10.1109/CLEI52000.2020.00034

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten