Skip to main content
SpringerLink
Log in

Toward the characterization of software testing practices in South America: looking at Brazil and Uruguay

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Software testing is an important activity in the software development life cycle. Several previous studies reported the results of surveys on software testing practices among practitioners from different countries. In this paper, we analyze these surveys aiming to get their main questions, and replicate a survey with practitioners from Brazil and Uruguay, two emerging South American software development scenarios. This survey was previously conducted in Manaus/Brazil in 2006 and Buenos Aires/Argentina in 2013. The replication’s scope includes three regions (Northern Brazil, Southern Brazil, and Uruguay). A total of 150 software testing practitioners responded to the survey. Its results are compared with the previous executions and other software testing surveys identified in the technical literature, strengthening previous findings. The Brazilian and Uruguayan participants indicate that: (1) documentation of test artifacts (plan, cases, procedures, results) are useful and important for software testing practitioners; (2) system and regression testing are the two test types deemed most useful and important; (3) tools for monitoring and management of test process tasks and bug reports are considered useful and important; (4) it is usual for software companies to have a definition of a testing process and to have a dedicated testing team; (5) there is a lack of measurement of test tasks and coverage in the industry; and (6) tools to support automation of test case generation and execution or code coverage are still poorly used in their organizations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Softex monitors and provides evidence of the performance variation of organizations adopting the MPS-SW, the Brazilian Software Process model—http://www.softex.br/mpsbr/performance-results/.

  2. https://www.limesurvey.org.

References

  • Ambler, S. W. (2009). Dr. Dobb’s Agile Update 01/09. Dr. Dobbs, v. January.

  • Andersson, C., & Runeson, P. (2002). Verification and validation in industry: A qualitative survey on the state of practice. In Proceedings of the 2002 international symposium on empirical software engineering (ISESE ‘02) (p. 37). IEEE Computer Society, Washington, DC.

  • Basili, V. R., Caldiera, G., & Rombach, H. D. (1994). The goal question metric approach. Encyclopedia of Software Engineering, 2, 528–532.

    Google Scholar 

  • Bertolino, A. (2004). The (Im)maturity level of software testing. ACM SIGSOFT Software Engineering Notes, 29(5), 1.

    Article  Google Scholar 

  • Camargo, K. G., Ferrari, F. C., & Fabbri, S. C. (2015). Characterizing the state of the practice in software testing through a TMMi-based process. Journal of Software Engineering Research and Development, 3(1), 7.

    Article  Google Scholar 

  • Causevic, A., Sundmark, D., & Punnekkat, S. (2010). An industrial survey on contemporary aspects of software testing. In ICST 2010: 3rd international conference on software testing, verification, and validation (pp. 393–401).

  • Chan, F. T., Tse, T. H., Tang, W. H., & Chen, T. Y. (2005). Software testing education and training in Hong Kong. In Proceedings of the fifth international conference on quality software (QSIC ‘05) (pp. 313-316). IEEE Computer Society, Washington, DC. doi:10.1109/QSIC.2005.57.

  • Consulting Capgemini Technology. (2016). World Quality Report 2015–2016. https://www.capgemini.com/thought-leadership/world-quality-report-2015-16. Accessed on 20 June 2016.

  • Craig, R., & Jaskiel, S. (2002). Systematic software testing. Boston: Artech House.

    MATH  Google Scholar 

  • Daka, E., & Fraser, G. (2014). A survey on unit testing practices and problems. In Proceedings of the 2014 IEEE 25th international symposium on software reliability engineering (ISSRE ‘14) (pp. 201–211). IEEE Computer Society, Washington, DC. doi:10.1109/ISSRE.2014.11.

  • Deak, A. (2014). A comparative study of testers’ motivation in traditional and agile software development. In Product-Focused Software Process Improvement (pp. 1–16).

  • Deak, A., & Stalhane, T. (2013). Organization of testing activities in Norwegian software companies. In Proceedings: IEEE 6th international conference on software testing, verification and validation workshops, ICSTW 2013 (pp. 102–107).

  • Deak, A., Stålhane, T., & Cruzes, D. (2013). Factors influencing the choice of a career in software testing among Norwegian students. In IASTED multiconferences: proceedings of the IASTED international conference on software engineering, SE 2013 (p. 796). ACTA Press, Calgary. doi:10.2316/P.2013.796-032.

  • Dias-Neto, A. C., Natali, A. C. C., Rocha, A., & Travassos, G. (2006). Caracterização do estado da prática das atividades de teste em um cenário de desenvolvimento de software brasileiro. In V SBQS (Simpósio Brasileiro de Qualidade de Software), Vila Velha, ES. http://www.lbd.dcc.ufmg.br/colecoes/sbqs/2006/002.pdf. Accessed on 20 June 2016.

  • Durelli, V. H. S., Araujo, R. F., Silva, M. A. G., et al. (2013). A scoping study on the 25 years of research into software testing in Brazil and an outlook on the future of the area. Journal of Systems and Software, 86(4), 934–950.

    Article  Google Scholar 

  • Engström, E., & Runeson, P. (2010). A qualitative survey of regression testing practices. In Proceedings of the 11th international conference on product-focused software process improvement (PROFES’10) (pp. 3–16). Springer, Berlin, Heidelberg. doi:10.1007/978-3-642-13792-1_3.

  • Garousi, V., Coşkunçay, A., Betin-Can, A., & Demirörs, O. (2015). A survey of software engineering practices in Turkey. Journal of Systems and Software, 108, 148–177.

    Article  Google Scholar 

  • Garousi, V., & Varma, T. (2010). A replicated survey of software testing practices in the Canadian province of Alberta: What has changed from 2004 to 2009? Journal of Systems and Software, 83(11), 2251–2262.

    Article  Google Scholar 

  • Garousi, V., & Zhi, J. (2013). A survey of software testing practices in Canada. Journal of Systems and Software, 86(5), 1354–1376.

    Article  Google Scholar 

  • Gelperin, D., & Hetzel, B. (1988). The growth of software testing. Communications of the ACM, 31(6), 687–695.

    Article  Google Scholar 

  • Geras, A. M., Smith, M. R., & Miller, J. (2004). A survey of software testing practices in Alberta. Canadian Journal of Electrical and Computer Engineering, 29(3), 183–191.

    Article  Google Scholar 

  • Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Chicago: Aldine Pub. v. 5.

    Google Scholar 

  • Greca, F., De Rossi, B., Robiolo, G., & Travassos, G. H. (2015). Aplicación y Valoración de la Verificación y Validación de Software: una Encuesta Realizada en Buenos Aires. In ASSE 2015: Simposio Argentino de Ingeniería de Software (pp. 26–40). Sociedad Argentina de Informática e Investigación Operativa (SADIO). ISSN: 2451-7593.

  • Greiler, M., Van Deursen, A., & Storey, M.-A. (2012). Test Confessions: A study of testing practices for plug-in systems. Proceedings of the 34th international conference on software engineering (ICSE ‘12) (pp. 244–254). Piscataway, NJ: IEEE Press.

    Google Scholar 

  • Grindal, M., Offutt, J., & Mellin, J. (2006). On the testing maturity of software producing organizations. In: Testing: Academic and Industrial Conference—Practice And Research Techniques (TAIC PART’06), IEEE (pp. 171–180). Windsor. doi:10.1109/TAIC-PART.2006.20.

  • IEEE. (1990). IEEE STD 610.12-1990. Standard glossary of software engineering terminology. [Institute of Electrical and Electronics Engineers, Ed.]IEEE STD 610.12-1990.

  • IEEE Standard. (1998). Standard for software test documentation. IEEE Std 829-1998.

  • ISTQB. (2015). ISTWB(R) Worldwide software testing practices report. http://www.istqb.org/references/surveys/istqb-worldwide-software-testing-practices-report.html. Accessed on 9 Mar.

  • Juristo, N., & Gómez, O. S. (2012). Replication of software engineering experiments. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). v. 7007 (pp. 60–88).

  • Juristo, N., Moreno, A. M., & Vegas, S. (2004). Reviewing 25 years of testing technique experiments. Empirical Software Engineering, 9(1/2), 7–44.

    Article  Google Scholar 

  • Kanij, T., Merkel, R., & Grundy, J. (2014). A preliminary survey of factors affecting software testers. In 2014 23rd Australian Software Engineering Conference, IEEE (pp. 180–189). Milsons Point, NSW. doi:10.1109/ASWEC.2014.32.

  • Kasurinen, J., Taipale, O., & Smolander, K. (2010). Software test automation in practice: Empirical observations. Advances in Software Engineering, 2010, 1–18.

    Article  Google Scholar 

  • Kirk, D., & Tempero, E. (2012). Software development practices in New Zealand. In 2012 19th Asia-Pacific Software Engineering Conference, IEEE (pp. 386–395). Hong Kong. doi:10.1109/APSEC.2012.51.

  • Lee, J., Kang, S., & Lee, D. (2012). Survey on software testing practices. IET Software, 6(3), 275.

    Article  Google Scholar 

  • Linaker, J., Sulaman, S. M., Höst, M., & De Melo, R. M. (2015). Guidelines for conducting surveys in software engineering. https://lup.lub.lu.se/search/publication/5366801. Accessed on 21 Mar.

  • Ng, S. P., Murnane, T., Reed, K., Grant, D., & Chen, T. Y. (2004). A preliminary survey on software testing practices in Australia. In Proceedings of the 2004 Australian software engineering conference (ASWEC ‘04) (p. 116). IEEE Computer Society, Washington, DC.

  • Pfahl, D., Yin, H., Mäntylä, M. V., & Münch, J. (2014). How is exploratory testing used? A state-of-the-practice survey. In Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM ‘14) (p. 10). ACM, New York, NY, Article 5. doi:10.1145/2652524.2652531.

  • Pham, R., Singer, L., Liskin, O., Filho, F. F., & Schneider, K. (2013). Creating a shared understanding of testing culture on a social coding site. Proceedings of the 2013 international conference on software engineering (ICSE ‘13) (pp. 112–121). Piscataway, NJ: IEEE Press.

    Chapter  Google Scholar 

  • Pressman, R., & Maxim, B. (2015). Software engineering: A practitioner’s approach. New York, NY: McGraw-Hill Education.

    Google Scholar 

  • Rafi, D. M., Moses, K., Petersen, K., & Mantyla, M. V. (2012). Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. Proceedings of the 7th international workshop on automation of software test (AST ‘12) (pp. 36–42). Piscataway, NJ: IEEE Press.

    Chapter  Google Scholar 

  • Runeson, P. (2006). A survey of unit testing practices. IEEE Software, 23(4), 22–29.

    Article  Google Scholar 

  • SOFTEX. (2012). Software and IT services. http://j.mp/Softex2012. Accessed on 20 June 2016.

  • Sung, P. W.-B., & Paynter, J. (2006). Software testing practices in New Zealand. In Proceedings of the 19th annual conference of the national advisory committee on computing qualifications (pp. 273–282).

  • Taipale, O., Smolander, K., & Kälviäinen, H. (2005). Finding and ranking research directions for software testing. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). v. 3792 (pp. 39–48).

  • Torkar, R., & Mankefors, S. (2003). A survey on testing and reuse. In IEEE international conference on software: Science, technology and engineering (SwSTE’03), IEEE (pp. 164–173). doi:10.1109/SWSTE.2003.1245437.

  • Travassos, G. H., & Kalinowski, M. (2012). iMPS 2012. Evidence on performance of organizations that adopted the MPS-SW Model since 2008. In SOFTEX: Association for promoting the Brazilian software excellence. www.softex.br/wp-content/uploads/2013/08/iMPS-2012.pdf. Accessed on 20 June 2016.

  • Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering (EASE ‘14) (p. 10). ACM, New York, NY. Article 38. doi:10.1145/2601248.2601268.

  • Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., & Wesslen, A. (2000). Experimentation in software engineering: an introduction. Norwell, MA: Kluwer Academic Publishers.

    Book  MATH  Google Scholar 

  • Wojcicki, M. A., & Strooper, P. (2006). A state-of-practice questionnaire on verification and validation for concurrent programs. In Proceedings of the 2006 workshop on Parallel and distributed systems: testing and debugging (PADTAD ‘06) (pp. 1–10). ACM, New York, NY. doi:10.1145/1147403.1147405.

Download references

Acknowledgments

This study could not be possible without the effective participation of practitioners. The authors would like to thank CNPq—CAcTUS Project (Grant 484380/2013-3), FAPEAM, and Universidad Austral for supporting this research. Prof. Travassos is a Brazilian CNPq Researcher (Grant 305929/2014-3). We also like to thank the anonymous reviewers for their suggestion of including the industry-wide section of this paper.

Author information

Authors and Affiliations

  1. Instituto de Computação, Universidade Federal do Amazonas (UFAM), Manaus, AM, Brazil

    Arilo Claudio Dias-Neto

  2. Facultad de Ingeniería, Universidad ORT Uruguay, Montevideo, Uruguay

    Santiago Matalonga & Martín Solari

  3. Universidad Austral, Buenos Aires, Argentina

    Gabriela Robiolo

  4. COPPE, Universidade Federal do Rio de Janeiro (COPPE/UFRJ), Rio de Janeiro, RJ, Brazil

    Guilherme Horta Travassos

Authors
  1. Arilo Claudio Dias-Neto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Santiago Matalonga
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martín Solari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gabriela Robiolo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guilherme Horta Travassos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arilo Claudio Dias-Neto.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dias-Neto, A.C., Matalonga, S., Solari, M. et al. Toward the characterization of software testing practices in South America: looking at Brazil and Uruguay. Software Qual J 25, 1145–1183 (2017). https://doi.org/10.1007/s11219-016-9329-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9329-3

Keywords

Search

Navigation