Lucas Santos
ABSTRACT
This paper presents the execution of a software testing experiment, where participants were tasked with identifying test cases from a provided software system and implementing the identified test cases. The objective of this study was to investigate students’ perception in test case creation. Participants were divided into two groups: a control group, which received the source code and documentation, and an experimental group, which received, in addition to the source code and documentation, a test suite generated by the EvoSuite tool. Furthermore, participants responded to three questionnaires: one about the project, another about their prior knowledge profile, and one providing feedback on their perception of the experiment and software testing education. Participants made it clear that the EvoSuite tool helped uncover new test scenarios. Additionally, it assisted in identifying new ways to test code segments that participants previously had no prior knowledge of how to access. Overall, the study suggests that the use of code generation tools can be a valuable addition to software testing education, aiding students in developing their testing skills and improving the quality of their tests. This work can benefit both teachers and students in computer science-related courses.
- P. Ammann and J. Offutt. 2016. Introduction to Software Testing (2 ed.). Cambridge University Press. https://doi.org/10.1017/9781316771273Google Scholar
Cross Ref
- M. Aniche, F. Hermans, and A. van Deursen. 2019. Pragmatic Software Testing Education. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. https://doi.org/10.1145/3287324.3287461Google ScholarDigital Library
- V. Basili and H. Dieter Rombach. 1988. The TAME Project: Towards Improvement-oriented Software Environments. IEEE Transactions on Software Engineering (TSE) (1988), 758–773.Google ScholarDigital Library
- F. B. V. Benitti. 2018. A Methodology to Define Learning Objects Granularity: A Case Study in Software Testing. Informatics Educ. 17 (2018), 1–20.Google ScholarCross Ref
- L. Bijlsma, N. Doorn, H. Passier, H. Pootjes, and S. Stuurman. 2021. How Do Students Test Software Units?. In Proceedings of the 43rd International Conference on Software Engineering: Joint Track on Software Engineering Education and Training. https://doi.org/10.1109/ICSE-SEET52601.2021.00029Google ScholarDigital Library
- C. K. D. Corte, A. C. Riestin, M. A. G. Silva, E. F. Barbosa, and Jose Carlos Maldonado. 2007. PROGTEST: Ambiente para submissão e avaliação de trabalhos práticos. In XVIII Simpósio Brasileiro de Informática na Educação (SBIE 2007) Workshop sobre Ambientes de Apoio à Aprendizagem de Algoritmos e Programação. São Paulo, SP.Google Scholar
- S. H. Edwards. 2004. Using Software Testing to Move Students from Trial-and-Error to Reflection-in-Action. SIGCSE Bull. 36, 1 (mar 2004), 26–30. https://doi.org/10.1145/1028174.971312Google ScholarDigital Library
- F. Ferreira, G. Vale, J. P. Diniz, and E. Figueiredo. 2020. On the Proposal and Evaluation of a Test-Enriched Dataset for Configurable Systems. In Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems (Magdeburg, Germany) (VaMoS ’20). Association for Computing Machinery, New York, NY, USA, Article 16, 10 pages. https://doi.org/10.1145/3377024.3377045Google ScholarDigital Library
- T. Ferreira, D. Viana, J. Fernandes, and R. Santos. 2018. Identifying Emerging Topics and Difficulties in Software Engineering Education in Brazil. In Proceedings of the XXXII Brazilian Symposium on Software Engineering. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3266237.3266247Google ScholarDigital Library
- G. Fraser and A. Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (Szeged, Hungary) (ESEC/FSE ’11). Association for Computing Machinery, New York, NY, USA, 416–419. https://doi.org/10.1145/2025113.2025179Google ScholarDigital Library
- V. Garousi, M. Felderer, M. Kuhrmann, and K. Herkiloğlu. 2017. What Industry Wants from Academia in Software Testing? Hearing Practitioners’ Opinions. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (Karlskrona, Sweden) (EASE’17). Association for Computing Machinery, New York, NY, USA, 65–69. https://doi.org/10.1145/3084226.3084264Google ScholarDigital Library
- V. Garousi, A. Rainer, P. Lauvås, and A. Arcuri. 2020. Software-testing education: A systematic literature mapping. Journal of Systems and Software 165 (2020), 110570. https://doi.org/10.1016/j.jss.2020.110570Google ScholarCross Ref
- A. Goffi, A. Gorla, M. D. Ernst, and M. Pezzè. 2016. Automatic Generation of Oracles for Exceptional Behaviors. In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016). Association for Computing Machinery, New York, NY, USA, 213–224. https://doi.org/10.1145/2931037.2931061Google ScholarDigital Library
- B. Gopal, S. Cooper, J. Olmanson, and R. Bockmon. 2021. Student Difficulties in Unit Testing, Integration Testing, and Continuous Integration: An Exploratory Pilot Qualitative Study. Psychology of Programming Interest Group (2021).Google Scholar
- D. Huizinga and A. Kolawa. 2007. Automated Defect Prevention: Best Practices in Software Management.Google Scholar
- A. Joshi, S. Kale, S. Chandel, and D. Pal. 2015. Likert Scale: Explored and Explained. British Journal of Applied Science & Technology 7 (01 2015), 396–403. https://doi.org/10.9734/BJAST/2015/14975Google ScholarCross Ref
- M. Leppänen, S. Mäkinen, M. Pagels, V.-P. Eloranta, J. Itkonen, M. V. Mäntylä, and T. Männistö. 2015. The highways and country roads to continuous deployment. IEEE Software 32, 2 (Mar 2015), 64–72. https://doi.org/10.1109/MS.2015.50Google ScholarDigital Library
- N. Lojo and A. Fox. 2022. Teaching Test-Writing As a Variably-Scaffolded Programming Pattern. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1 (Dublin, Ireland) (ITiCSE ’22). Association for Computing Machinery, New York, NY, USA, 498–504. https://doi.org/10.1145/3502718.3524789Google ScholarDigital Library
- J. Mendes, Y. Costa, K. Frazão, R. Santos, D. Santos, and L. Rivero. 2019. Identificação das Expectativas e Dificuldades de Alunos de Graduação no Ensino de Engenharia de Software. In Anais do XXVII Workshop sobre Educação em Computação (Belém). SBC, Porto Alegre, RS, Brasil, 334–347. https://doi.org/10.5753/wei.2019.6640Google ScholarCross Ref
- G. G. Moreira, S. Reinehr, A. Malucelli, and F. Van Amstel. 2023. ProTesters: A Board Game for Teaching the Testing Process. In Proceedings of the XXI Brazilian Symposium on Software Quality (Curitiba, Brazil) (SBQS ’22). Association for Computing Machinery, New York, NY, USA, Article 28, 9 pages. https://doi.org/10.1145/3571473.3571503Google ScholarDigital Library
- E. Oliveira Jr, T. Colanzi, A. Amaral, A. Cordeiro, J. C. Neto, and S. Souza. 2021. Ensino, Aprendizagem e Uso Profissional da UML em Maringá e Região. In Anais do XXIX Workshop sobre Educação em Computação (Evento Online). SBC, Porto Alegre, RS, Brasil, 328–337. https://doi.org/10.5753/wei.2021.15924Google ScholarCross Ref
- C. Pacheco and M. D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java. In OOPSLA 2007 Companion, Montreal, Canada. ACM.Google ScholarDigital Library
- L. Paschoal and S. Souza. 2018. A Survey on Software Testing Education in Brazil. https://doi.org/10.1145/3275245.3275289Google ScholarDigital Library
- G. Petrović, M. Ivanković, G. Fraser, and R. Just. 2022. Practical Mutation Testing at Scale: A view from Google. IEEE Transactions on Software Engineering 48, 10 (2022), 3900–3912. https://doi.org/10.1109/TSE.2021.3107634Google ScholarDigital Library
- N. Setiani, R. Ferdiana, P. I. Santosa, and R. Hartanto. 2019. Literature Review on Test Case Generation Approach. In Proceedings of the 2nd International Conference on Software Engineering and Information Management (Bali, Indonesia) (ICSIM 2019). Association for Computing Machinery, New York, NY, USA, 91–95. https://doi.org/10.1145/3305160.3305186Google ScholarDigital Library
- M. Shahin, M. Ali Babar, and L. Zhu. 2017. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 5 (2017), 3909–3943. https://doi.org/10.1109/ACCESS.2017.2685629Google ScholarCross Ref
- M. Silva, T. Rocha, and M. Perkusich. 2021. O ensino de engenharia de software no nível superior: um mapeamento sistemático. Revista Principia - Divulgação Científica e Tecnológica do IFPB 1 (04 2021). https://doi.org/10.18265/1517-0306a2021id3874Google ScholarCross Ref
- D. Souza, J. C. Maldonado, and E. F. Barbosa. 2012. Aspectos de Desenvolvimento e Evolução de um Ambiente de Apoio ao Ensino de Programação e Teste de Software. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE) 23, 1 (2012). https://doi.org/10.5753/cbie.sbie.2012.%pGoogle ScholarCross Ref
- A. B. Sánchez, P. Delgado-Pérez, I. Medina-Bulo, and S. Segura. 2022. Mutation Testing in the Wild: Findings from GitHub. Empirical Softw. Engg. 27, 6 (nov 2022), 35 pages. https://doi.org/10.1007/s10664-022-10177-8Google ScholarDigital Library
- P. H. Valle, E. Barbosa, and J. Maldonado. 2015. CS curricula of the most relevant universities in Brazil and abroad: Perspective of software testing education. 62–68. https://doi.org/10.1109/SIIE.2015.7451649Google ScholarCross Ref
- S. Vogl, S. Schweikl, G. Fraser, A. Arcuri, J. Campos, and A. Panichella. 2021. EVOSUITE at the SBST 2021 Tool Competition. In 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST). IEEE, 28–29.Google Scholar
- Claes W.2014. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE). 1–10.Google Scholar
- H. A. P. Zanetti, M. A. F. Borges, and I. L. M. Ricarte. 2023. ComFAPOO: Método de Ensino de Programação Orientada à Objetos Baseado em Aprendizagem Significativa e Computação Física. Revista Brasileira de Informática na Educação 31 (jan. 2023), 01–30. https://doi.org/10.5753/rbie.2023.2851Google ScholarCross Ref
Index Terms
- An experiment on software test creation: can EvoSuite help test teaching?
Index terms have been assigned to the content through auto-classification.
Recommendations
A detailed investigation of the effectiveness of whole test suite generation
A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., lines, branches, mutants). Rather than generating one test case at a time for each of these goals individually, whole ...
Empirical Study of Effectiveness of EvoSuite on the SBST 2020 Tool Competition Benchmark
Search-Based Software EngineeringAbstractEvoSuite is a state-of-the-art search-based software testing tool for Java programs and many researchers have applied EvoSuite to achieve high test coverage. However, due to high complexity of object-oriented programs, EvoSuite still suffers ...
Mock object creation for test factoring
PASTE '04: Proceedings of the 5th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineeringTest factoring creates fast, focused unit tests from slow system-wide tests; each new unit test exercises only a subset of the functionality exercised by the system tests. Augmenting a test suite with factored unit tests, and prioritizing the tests, ...
Comments