Davi Monteiro
ABSTRACT
Empirical evaluations developed in the software engineering area have been widely applied as a formalism to validate and ensure the credibility of the works proposed by the researchers. Even though the adoption of empirical evaluation techniques has gained popularity in recent years, its application has been questioned both qualitatively and quantitatively. This study aims at analyzing how empirical software engineering research has evolved in the Brazilian Symposium on Software Engineering (SBES) community. We performed a controlled quasi-experiment, using published papers over the last 10 years in SBES. Our experiment was divided into two phases: classification by type and quality assessment of the main empirical types. In the first phase, the sample was 201 papers; in the second one, the sample decreased to 126 papers. The results have shown failures and gaps in the application of empirical methods when assessing the quality of the Software Engineering works. We believe that we can contribute to improve how the studies were conducted and consequently help to produce more reliable results, reducing or eliminating biases: an important qualitative factor in scientific work. In addition, due to the lack of assessment supporting tools, we developed a theoretical protocol to support the assessment process and proposed improvements for papers that obtained below-expected rates.
- Victor R. Basili. 1996. The Role of Experimentation in Software Engineering: Past, Current, and Future. In Proceedings of the 18th International Conference on Software Engineering (ICSE). IEEE Computer Society, 442--449. Google Scholar
Digital Library
- Jeffrey C. Carver, Eugene Syriani, and Jeff Gray. 2011. Assessing the Frequency of Empirical Evaluation in Software Modeling Research.. In Proceedings of the First International Workshop on Experiences and Empirical Studies in Software Modelling. CEUR-WS.org, 28--37.Google Scholar
- A. Fink. 2003. The Survey Handbook. SAGE Publications.Google Scholar
- G.V. Glass, B. McGaw, and M.L. Smith. 1981. Meta-analysis in social research. Sage Publications.Google Scholar
- Staffs Keele. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE Technical Report. EBSE. sn.Google Scholar
- Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, and Stephen Linkman. 2009. Systematic Literature Reviews in Software Engineering - A Systematic Literature Review. Information and Software Technology 51, 1 (Jan. 2009), 7--15. Google ScholarDigital Library
- Barbara A. Kitchenham, O. Pearl Brereton, David Budgen, and Zhi Li. 2009. An Evaluation of Quality Checklist Proposals: A Participant-observer Case Study. In Proceedings of the 13th International Conference on Evaluation and Assessment in Software Engineering (EASE'09). BCS Learning & Development Ltd., Swindon, UK, 55--64. http://dl.acm.org/citation.cfm?id=2227040.2227047 Google ScholarDigital Library
- Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2015. Evidence-Based Software Engineering and Systematic Reviews. Chapman & Hall/CRC. Google ScholarDigital Library
- Barbara A Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-based software engineering. In Proceedings of the 26th international conference on software engineering. IEEE Computer Society, 273--281. Google ScholarDigital Library
- Robert V. Labaree. 2017. Research Methods in the Social Sciences. (June 2017). http://lynn-library.libguides.com/c.php?g=549455&p=3771805Google Scholar
- Johan Linåker, Sardar Muhammad Sulaman, Rafael Maiani de Mello, and Martin Höst. 2015. Guidelines for conducting surveys in software engineering. http://portal.research.lu.se/portal/files/6062997/5463412.pdf. (2015). Accessed 10 December 2016.Google Scholar
- Ruchika Malhotra. 2015. Empirical Research in Software Engineering: Concepts, Analysis, and Applications. Chapman & Hall/CRC. Google ScholarDigital Library
- Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J Ware, and John PA Ioannidis. 2017. A manifesto for reproducible science. Nature Human Behaviour 1 (2017), 0021.Google ScholarCross Ref
- Dewayne E. Perry, Susan Elliott Sim, and Steve M. Easterbrook. 2004. Case Studies for Software Engineers. In Proceedings of the 26th International Conference on Software Engineering (ICSE '04). IEEE Computer Society, Washington, DC, USA, 736--738. Google ScholarDigital Library
- Per Runeson and Martin Höst. 2009. Guidelines for Conducting and Reporting Case Study Research in Software Engineering. Empirical Software Engineering 14, 2 (April 2009), 131--164. Google ScholarDigital Library
- Per Runeson, Martin Host, Austen Rainer, and Bjorn Regnell. 2012. Case Study Research in Software Engineering: Guidelines and Examples. Wiley Publishing. Google ScholarDigital Library
- Mary Shaw. 2003. Writing good software engineering research papers. In Proceedings of the 25th International Conference on Software Engineering (ICSE). IEEE, 726--736. Google ScholarDigital Library
- Paulo Anselmo Da Mota Silveira Neto, JoáS Sousa Gomes, Eduardo Santana De Almeida, Jair Cavalcanti Leite, Thais Vasconcelos Batista, and Larissa Leite. 2013. 25 Years of Software Engineering in Brazil: Beyond an Insider's View. J. Syst. Softw. 86, 4 (April 2013), 872--889. Google ScholarDigital Library
- Dag I. K. Sjoberg, Tore Dyba, and Magne Jorgensen. 2007. The Future of Empirical Methods in Software Engineering Research. In Proceedings of the Future of Software Engineering (FOSE). IEEE Computer Society, 358--378. Google ScholarDigital Library
- Andrs Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101--132.Google Scholar
- Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media. Google ScholarCross Ref
- R.K. Yin. 2009. Case Study Research: Design and Methods. SAGE Publications.Google Scholar
- Carmen Zannier, Grigori Melnik, and Frank Maurer. 2006. On the Success of Empirical Studies in the International Conference on Software Engineering. In Proceedings of the 28th International Conference on Software Engineering (ICSE '06). ACM, New York, NY, USA, 341--350. Google ScholarDigital Library
Index Terms
- An Analysis of the Empirical Software Engineering over the last 10 Editions of Brazilian Software Engineering Symposium
Recommendations
Issues in applying empirical software engineering to software architecture
ECSA'07: Proceedings of the First European conference on Software ArchitectureEmpirical software engineering focuses on the evaluation of software engineering technologies, such as processes and tools, by comparing related sets of data. It has contributed a valuable body of knowledge in several areas such as Software Economics ...
Empirical software engineering for agent programming
AGERE! 2012: Proceedings of the 2nd edition on Programming systems, languages and applications based on actors, agents, and decentralized control abstractionsEmpirical software engineering is a branch of software engineering in which empirical methods are used to evaluate and develop tools, languages and techniques. In this position paper we argue for the use of empirical methods to advance the area of agent ...
Applying empirical software engineering to software architecture: challenges and lessons learned
In the last 15 years, software architecture has emerged as an important software engineering field for managing the development and maintenance of large, software-intensive systems. Software architecture community has developed numerous methods, ...
Comments