Abstract
The systematic review (SR) is a methodology used to find and aggregate all relevant existing evidence about a specific research question of interest. One of the activities associated with the SR process is the selection of primary studies, which is a time consuming manual task. The quality of primary study selection impacts the overall quality of SR. The goal of this paper is to propose a strategy named “Score Citation Automatic Selection” (SCAS), to automate part of the primary study selection activity. The SCAS strategy combines two different features, content and citation relationships between the studies, to make the selection activity as automated as possible. Aiming to evaluate the feasibility of our strategy, we conducted an exploratory case study to compare the accuracy of selecting primary studies manually and using the SCAS strategy. The case study shows that for three SRs published in the literature and previously conducted in a manual implementation, the average effort reduction was 58.2 % when applying the SCAS strategy to automate part of the initial selection of primary studies, and the percentage error was 12.98 %. Our case study provided confidence in our strategy, and suggested that it can reduce the effort required to select the primary studies without adversely affecting the overall results of SR.
Similar content being viewed by others
Notes
Instructions for using Weka and applying SCAS in StArt are available at http://www2.dc.ufscar.br/~lapes/SCAS/
References
Babar MA, Zhang H (2010) Systematic literature reviews in software engineering: Preliminary results from interviews with researchers. In: Proc, ESEM’09, 1–10
Boell S, Cezec-Kecmanovic D (2011) “Are systematic reviews better, less biased and of higher quality?” ECIS 2011 Proceedings. Paper 223. http://aisel.aisnet.org/ecis2011/223
Brereton OP, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(1):571–583
Carletta J (1996) Assessing agreement on classification tasks: the kappa statistic. Comput Linguis 22(2):249–254
Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. In: Proc. ESEM’09, 513–539
Dybå T, Kitchenham BA, Jorgensen M (2005) Evidence-based software engineering for practitioners. IEEE Softw 22(1):58–65
Dybå T, Dingsøyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: An experience report. In: Proc. ESEM’07, 225–234
Fabbri S, Hernandes E, Di Thommazo A, Belgamo A, Zamboni A, Silva C (2012) Managing literature reviews information through visualization. In: Proc. ICEIS’12, 36–45
Felizardo KR, Salleh N, Martins R, Mendes E, MacDonell S, Maldonado JC (2011) Using visual text mining to support the study selection activity in systematic literature reviews. In: Proc. ESEM’11, 1–10
Felizardo KR, Andery G, Paulovich F, Minghim R, Maldonado JC (2012) A visual analysis approach to validate the selection review of primary studies in systematic reviews. Inf Softw Technol 54(10):1079–1091
Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report, Keele University and University of Durham, version 2.3
Kitchenham BA, Dybå T, Jørgensen M (2004) Evidence-based software engineering. In: Proc. ICSE’08, 273–281
Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174
Malheiros V, Hohn E, Pinho R, Mendonca M, Maldonado JC (2007) A visual text mining approach for systematic reviews. In: Proc. ESEM’07, 245–254
Marshall C, Brereton P (2013) Tools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study. In: Proc. ESEM’13, 296–299
Oates BJ, Capper G (2009) Using systematic reviews and evidence-based software engineering with masters students. In: Proc. EASE’09, 1–8
Paulovich FV, Minghim R (2008) HiPP: a novel hierarchical point placement strategy and its application to the exploration of document collections. IEEE Trans Vis Comput Graph 14(6):1229–1236
Petersen K, Ali NB (2011) Identifying strategies for study selection in systematic reviews and maps. In: Proc. ESEM’11, 351–354
Riaz M, Sulayman M, Salleh N, Mendes E (2010) Experiences conducting systematic reviews from novices’ perspective. In: Proc. EASE’10, 1–10
Shaw M, Clements P (2006) The golden age of software architecture: A comprehensive survey. Technical Report CMU-ISRI-06-101, Software Engineering Institute, Carnegie Mellon University
Shepperd M (2007) Software project economics: A roadmap. In: Proc. FOSE’07, IEEE Computer Society, Washington, DC, pp 304–315
Weka (2013) Available at http://www.cs.waikato.ac.nz/ml/weka/index.html. Last access on 30-March-2013
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in Software Engineering: An Introduction. In: Kluwer Academic Publishers (ed), 1st edn. Boston, US, pp 63–74
Zhang H, Muhammad AB (2011) An empirical investigation of systematic reviews in software engineering. In: Proc. ESEM’11, 1–10
Acknowledgments
This research is supported by the Brazilian funding agency: FAPESP (Process no. 2012/02524-4).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Tony Gorschek
Rights and permissions
About this article
Cite this article
Octaviano, F.R., Felizardo, K.R., Maldonado, J.C. et al. Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?. Empir Software Eng 20, 1898–1917 (2015). https://doi.org/10.1007/s10664-014-9342-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-014-9342-8