Skip to main content
SpringerLink
Log in

Scientific Workflow Management for Software Quality Assessment Replication: An Open Source Architecture

  • Conference paper
  • First Online:
Quality of Information and Communications Technology (QUATIC 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1871))

  • 196 Accesses

Abstract

Replication of research experiments is important for establishing the validity and generalizability of findings, building a cumulative body of knowledge, and addressing issues of publication bias. The quest for replication led to the concept of scientific workflow, a structured and systematic process for carrying out research that defines a series of steps, methods, and tools needed to collect and analyze data, and generate results.

In this study, we propose a cloud-based framework built upon open source software, which facilitates the construction and execution of workflows for the replication/reproduction of software quality studies. To demonstrate its feasibility, we describe the replication of a software quality experiment on automatically detecting code smells with machine learning techniques.

The proposed framework can mitigate two types of validity threats in software quality experiments: (i) internal validity threats due to instrumentation, since the same measurement instruments can be used in replications, thus not affecting the validity of the results, and (ii) external validity threats due to reduced generalizability, since different researchers can more easily replicate experiments with different settings, populations, and contexts while reusing the same scientific workflow.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    This application is developed by the Institute of Computer Science at the Johannes Gutenberg University Mainz, inserted in the openrisk project.

  2. 2.

    https://studio.azureml.net/.

  3. 3.

    https://www.cs.waikato.ac.nz/ml/index.html.

  4. 4.

    https://rapidminer.com/.

  5. 5.

    http://orange.biolab.si/.

  6. 6.

    https://www.h2o.ai/.

  7. 7.

    https://www.cs.waikato.ac.nz/ml/weka/arff.html.

References

  1. Abbuhl, R.: Why, when, and how to replicate research. In: Research Methods in Second Language Acquisition: A Practical Guide, pp. 296–312 (2012). https://doi.org/10.1002/9781444347340.ch15

  2. Bryton, S., Brito e Abreu, F., Monteiro, M.: Reducing subjectivity in code smells detection: experimenting with the Long Method. In: Proceedings of the 7th International Conference on the Quality of Information and Communications Technology (QUATIC), pp. 337–342. IEEE (2010). https://doi.org/10.1109/QUATIC.2010.60

  3. Cohen-Boulakia, S., Chen, J., Missier, P., Goble, C., Williams, A.R., Froidevaux, C.: Distilling structure in taverna scientific workflows: a refactoring approach. BMC Bioinformatics 15(Suppl 1), 1–14 (2014). https://doi.org/10.1186/1471-2105-15-S1-S12

    Article  Google Scholar 

  4. De Magalhães, C.V., Da Silva, F.Q., Santos, R.E., Suassuna, M.: Investigations about replication of empirical studies in software engineering: a systematic mapping study. Inf. Softw. Technol. 64, 76–101 (2015). https://doi.org/10.1016/j.infsof.2015.02.001

    Article  Google Scholar 

  5. De Roure, D., Goble, C., Stevens, R.: The design and realisation of the Experimentmy virtual research environment for social sharing of workflows. Futur. Gener. Comput. Syst. 25(5), 561–567 (2009). https://doi.org/10.1016/j.future.2008.06.010

    Article  Google Scholar 

  6. Deelman, E., et al.: The future of scientific workflows. Int. J. High-Perform. Comput. Appl. 32(1), 159–175 (2018). https://doi.org/10.1177/1094342017704893

    Article  Google Scholar 

  7. Deelman, E., et al.: Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst. 46, 17–35 (2015). https://doi.org/10.1016/J.FUTURE.2014.10.008

    Article  Google Scholar 

  8. Fokaefs, M., Tsantalis, N., Stroulia, E.: JDeodorant: identification and application of extract class refactorings. In: Proceedings of the 33rd International Conference on Software Engineering, (ICSE). ACM/IEEE (2011). https://doi.org/10.1145/1985793.1985989

  9. Fontana, F.A., Mangiacavalli, M., Pochiero, D., Zanoni, M.: On experimenting refactoring tools to remove code smells. In: Proceedings of the XP’15 Workshops, pp. 1–8. ACM Press, New York (2015). https://doi.org/10.1145/2764979.2764986

  10. Fontana, F.A., Zanoni, M., Marino, A., Mäntylä, M.V.: Code smell detection: towards a machine learning-based approach. In: Proceedings of the International Conference on Software Maintenance (ICSM). IEEE (2013). https://doi.org/10.1109/ICSM.2013.56

  11. Gómez, O.S., Juristo, N., Vegas, S.: Understanding replication of experiments in software engineering: a classification. Inf. Softw. Technol. 56(8), 1033–1048 (2014). https://doi.org/10.1016/j.infsof.2014.04.004

    Article  Google Scholar 

  12. Harman, M., McMinn, P., de Souza, J.T., Yoo, S.: Search based software engineering: techniques, taxonomy, tutorial. In: Meyer, B., Nordio, M. (eds.) LASER 2008-2010. LNCS, vol. 7007, pp. 1–59. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-25231-0_1

    Chapter  Google Scholar 

  13. Ivie, P., Thain, D.: Reproducibility in scientific computing. ACM Comput. Surv. 51(3), 1–36 (2018). https://doi.org/10.1145/3186266

    Article  Google Scholar 

  14. Juristo, N., Gómez, O.S.: Replication of software engineering experiments. In: Meyer, B., Nordio, M. (eds.) Empirical Software Engineering and Verification. Lecture Notes in Computer Science, vol. 7007, pp. 60–88. Springer, Berlin (2012)

    Chapter  Google Scholar 

  15. Kitchenham, B.: The role of replications in empirical software engineering-a word of warning. Empir. Softw. Eng. 13(2), 219–221 (2008). https://doi.org/10.1007/s10664-008-9061-0

    Article  Google Scholar 

  16. La Sorte, M.A.: Replication as a verification technique in survey research: a paradigm. Sociol. Q. 13(2), 218–227 (1972). https://doi.org/10.1111/j.1533-8525.1972.tb00805.x

    Article  Google Scholar 

  17. Liu, H., Ma, Z., Shao, W., Niu, Z.: Schedule of bad smell detection and resolution: a new way to save effort. IEEE Trans. Softw. Eng. 38(1), 220–235 (2012). https://doi.org/10.1109/TSE.2011.9

    Article  Google Scholar 

  18. Mantyla, M., Lassenius, C.: Subjective evaluation of software evolvability using code smells: an empirical study. Empir. Softw. Eng. 11(3), 395–431 (2006). https://doi.org/10.1007/s10664-006-9002-8

    Article  Google Scholar 

  19. Mantyla, M., Vanhanen, J., Lassenius, C.: Bad smells - humans as code critics. In: Proceedings of the 20th International Conference on Software Maintenance (ICSM), pp. 399–408 (2004). https://doi.org/10.1109/ICSM.2004.1357825

  20. Palomba, F., Bavota, G., Penta, M.D., Oliveto, R., Poshyvanyk, D., Lucia, A.D.: Mining version histories for detecting code smells. IEEE Trans. Software Eng. 41(5), 462–489 (2015). https://doi.org/10.1109/TSE.2014.2372760

    Article  Google Scholar 

  21. Pessoa, T., Brito e Abreu, F., Monteiro, M.P., Bryton, S.: An eclipse plugin to support code smells detection. In: Proceedings of INFORUM 2011 (Simpósio de Informática). p. 12 (2011). https://arxiv.org/abs/1204.6492

  22. Pereira dos Reis, J., Brito e Abreu, F., de Figueiredo Carneiro, G., Anslow, C.: Code smells detection and visualization: a systematic literature review. Arch. Comput. Methods Eng. 29(1), 47–94 (2022). https://doi.org/10.1007/s11831-021-09566-x

  23. Shepperd, M.: Replication studies considered harmful. In: Proceedings of the International Conference on Software Engineering (ICSE), pp. 73–76. ACM/IEEE (2018). https://doi.org/10.1145/3183399.3183423

  24. Shull, F.J., Carver, J.C., Vegas, S., Juristo, N.: The role of replications in empirical software engineering. Empir. Softw. Eng. 13(2), 211–218 (2008). https://doi.org/10.1007/s10664-008-9060-1

    Article  Google Scholar 

  25. Taylor, I.J., Deelman, E., Gannon, D., Shields, M.S.: Workflows for E-science: Scientific Workflows for Grids. Springer, Cham (2007). https://doi.org/10.1007/978-1-84628-757-2

    Book  Google Scholar 

  26. Tsantalis, N., Chaikalis, T., Chatzigeorgiou, A.: JDeodorant: identification and removal of type-checking bad smells. In: Proceedings of the 12th European Conference on Software Maintenance and Reengineering (CSMR), pp. 329–331 (2008). https://doi.org/10.1109/CSMR.2008.4493342

  27. Wang, C., Hirasawa, S., Takizawa, H., Kobayashi, H.: Identification and elimination of platform-specific code smells in high performance computing applications. Int. J. Networking Comput. 5(1), 180–199 (2015). https://doi.org/10.15803/ijnc.5.1_180

    Article  Google Scholar 

  28. Wolstencroft, K., et al.: The taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud. Nucleic Acids Res. 41(Web Server issue), 557–561 (2013). https://doi.org/10.1093/nar/gkt328

  29. Wolstencroft, K., Fisher, P., Goble, C.: Scientific workflows overview. Connexions 26, 1–6 (2009)

    Google Scholar 

  30. Yamashita, A., Moonen, L.: To what extent can maintenance problems be predicted by code smell detection? - An empirical study. Inf. Softw. Technol. 55(12), 2223–2242 (2013). https://doi.org/10.1016/j.infsof.2013.08.002

    Article  Google Scholar 

Download references

Acknowledgement

This research was partially funded by ISTAR’s projects FCT UIDB/04466/2020 and UIDP/04466/2020.

Author information

Authors and Affiliations

  1. Instituto Superior de Tecnologias Avançadas (ISTEC), Lisboa, Portugal

    José Pereira dos Reis

  2. Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR-IUL, Lisboa, Portugal

    José Pereira dos Reis, Fernando Brito e Abreu & Duarte Almeida

  3. Federal University of Sergipe (UFS), Aracaju, Brazil

    Glauco de F. Carneiro

Authors
  1. José Pereira dos Reis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fernando Brito e Abreu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Glauco de F. Carneiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Duarte Almeida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to José Pereira dos Reis .

Editor information

Editors and Affiliations

  1. University of Aveiro, Aveiro, Portugal

    José Maria Fernandes

  2. Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

    Guilherme H. Travassos

  3. University of Oulu, Oulu, Finland

    Valentina Lenarduzzi

  4. University of Oulu, Oulu, Finland

    Xiaozhou Li

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

dos Reis, J.P., Abreu, F.B.e., de F. Carneiro, G., Almeida, D. (2023). Scientific Workflow Management for Software Quality Assessment Replication: An Open Source Architecture. In: Fernandes, J.M., Travassos, G.H., Lenarduzzi, V., Li, X. (eds) Quality of Information and Communications Technology. QUATIC 2023. Communications in Computer and Information Science, vol 1871. Springer, Cham. https://doi.org/10.1007/978-3-031-43703-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43703-8_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43702-1

  • Online ISBN: 978-3-031-43703-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation