Abstract
Context
The number of published systematic literature reviews (SLRs) in software engineering venues is increasing. However, even with their high adaptation rate, the task of performing an SLR requires a large amount of effort and presents a number of barriers. Specifically, during the SLR search phase authors must expend a lot of time and overcome a large number of barriers.
Objective
To help alleviate some of the barriers in the search phase, we identify and prioritize SLR search tool requirements based on input from the SLR community. These requirements will help tool builders ensure they focus their efforts appropriately.
Method
We conducted an SLR and a survey of SLR authors in software engineering. In the SLR we extracted problems and solutions SLR authors reported during their search processes. In the survey we asked respondents to describe the problems they faced during SLR search and to specify any requirements they would like to see as part of an SLR search tool. We also asked survey respondents to describe any tools they use to support SLR search, along with the strengths and limitations of those tools.
Results
Based on analysis of 84 studies from the literature and a qualitative analysis of 131 responses from researchers, we identified a set of functional requirements that authors, researchers, and tool builders can use as a reference. We also describe the tools currently used by SLR researchers.
Conclusions
The list of SLR search tool requirements can be used by tool builders as a guide when constructing new tools. Our analysis of tools showed that more recent tools are covering more of the requirements than in the past.



Similar content being viewed by others
Notes
References
Al-Zubidy A, Carver JC, Hale DP, Hassler EE (2017) Vision for {SLR} tooling infrastructure: prioritizing value-added requirements. Inf Softw Technol. https://doi.org/10.1016/j.infsof.2017.06.007
Allman-Farinelli M, Byron A, Collins C, Gifford J, Williams P (2014) Challenges and lessons from systematic literature reviews for the australian dietary guidelines. Aust J Prim Health 20(3):236–240
Babar MA, Zhang H (2009) Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In: 3rd international symposium on empirical software engineering and measurement. IEEE Computer Society, pp 346–355
Badampudi D, Wohlin C, Petersen K (2015) Experiences from using snowballing and database searches in systematic literature studies. In: 19th international conference on evaluation and assessment in software engineering. ACM, p 17
Barat S, Clark T, Barn B, Kulkarni V (2017) A model-based approach to systematic review of research literature. In: Proceedings of the 10th innovations in software engineering conference. ACM, pp 15–25
Barn B, Raimondi F, Athappian L, Clark T (2014) Slrtool: a tool to support collaborative systematic literature reviews. In: ICEIS 2014—Proceedings of the 16th international conference on enterprise information systems, vol 2, pp 440–447
Biolchini J, Mian PG, Natali ACC, Travassos GH (2005) Systematic review in software engineering. System Engineering and Computer Science Department COPPE/UFRJ. Technical Report ES 679(05):45
Bowes D, Hall T, Beecham S (2012) Slurp: a tool to help large complex systematic literature reviews deliver valid and rigorous results. In: Proceedings of the 2nd international workshop on evidential assessment of software technologies. ACM, pp 33–36
Brereton P, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583
Budgen D, Brereton P, Drummond S, Williams N (2018) Reporting systematic reviews: some lessons from a tertiary study. Inf Softw Technol 95:62–74. https://doi.org/10.1016/j.infsof.2017.10.017
Carver JC, Hassler E, Hernandes E, Kraft NA (2013) Identifying barriers to the systematic literature review process. In: 7th international symposium on empirical software engineering and measurement. IEEE, pp 203–212
Corbin J, Strauss A et al (2008) Basics of qualitative research: techniques and procedures for developing grounded theory. Thousand Oaks
de Almeida Biolchini JC, Mian PG, Natali ACC, Conte TU, Travassos GH (2007) Scientific research ontology to support systematic review in software engineering. Adv Eng Inf 21(2):133–151
Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. Empir Softw Eng 14(5):513–539
Dyba T, Dingsoyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: an experience report. In: 1st international symposium on empirical software engineering and measurement. IEEE, pp 225–234
Engström E, Skoglund M, Runeson P (2008) Empirical evaluations of regression test selection techniques: a systematic review. In: 1st international symposium on empirical software engineering and measurement. ACM, pp 22–31
Felizardo KR, Mendes E, Kalinowski M, Souza ÉF, Vijaykumar NL (2016) Using forward snowballing to update systematic reviews in soft. engineering. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement. ACM, p 53
Fernández-Sáez A M, Bocco MG, Romero FP (2010) Slr-tool: a tool for performing systematic literature reviews. In: ICSOFT (2), pp 157–166
Hassler E, Carver JC, Kraft NA, Hale D (2014) Outcomes of a community workshop to identify and rank barriers to the systematic literature review process. In: 18th International conference on evaluation & assessment in software engineering. ACM, p 31
Hassler E, Carver JC, Hale D, Al-Zubidy A (2016) Identification of SLR tool needs–results of a community workshop. J Inf Softw Technol 70:122–129
Hernandes E, Zamboni A, Fabbri S, Thommazo AD (2012) Using gqm and tam to evaluate start-a tool that supports systematic review. CLEI Electron J 15 (1):3–3
Jalali S, Wohlin C (2012) Systematic literature studies: database searches vs. backward snowballing. In: Proceedings of the ACM-IEEE international symposium on empirical software engineering and measurement. ACM, pp 29–38
Kitchenham B (2004) Procedures for performing systematic reviews. Tech. Rep. TR/SE-0401, Keel University
Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE-2007-01, Keel University
Kitchenham B, Linkman S, Law D (1997) Desmet: a methodology for evaluating software engineering methods and tools. Comput Control Eng J 8(3):120–126
Kitchenham BA, Dyba T, Jorgensen M (2004) Evidence-based software engineering. In: 26th international conference on software engineering. IEEE Computer Society, pp 273–281
Kitchenham B, Pretorius R, Budgen D, Brereton OP, Turner M, Niazi M, Linkman S (2010) Systematic literature reviews in software engineering–a tertiary study. Inf Softw Technol 52(8):792– 805
Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews, vol 4. CRC Press, Boca Raton
Mallett R, Hagen-Zanker J, Slater R, Duvendack M (2012) The benefits and challenges of using systematic reviews in international development research. J Dev Effect 4(3):445–455
Marshall C, Brereton P (2013) Tools to support systematic literature reviews in software engineering: a mapping study. In: 7th international symposium on empirical software engineering and measurement. IEEE, pp 296–299
Marshall C, Brereton P, Kitchenham B (2014) Tools to support systematic reviews in software engineering: a feature analysis. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. ACM, p 13
Molléri JS, Benitti FBV (2015) Sesra: a web-based automated tool to support the systematic literature review process. In: Proceedings of the 19th international conference on evaluation and assessment in software engineering. ACM, p 24
Parsifal (2015) Parsifal systematic literature review tool. [Online; accessed 1–December-2016]
Perestelo-Pérez L (2013) Standards on how to develop and report systematic reviews in psychology and health. Int J Clin Health Psychol 13(1):49–57. https://doi.org/10.1016/S1697-2600(13)70007-3
Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: EASE, vol 8, pp 68–77
Pignone M, Saha S, Hoerger T, Lohr KN, Teutsch S, Mandelblatt J (2005) Challenges in systematic reviews of economic analyses. Ann Intern Med 142 (12_Part_2):1073–1079
Reed D, Price EG, Windish DM, Wright SM, Gozu A, Hsu EB, Beach MC, Kern D, Bass EB (2005) Challenges in systematic reviews of educational intervention studies. Ann Intern Med 142(12_Part_2):1080–1089
Riaz M, Sulayman M, Salleh N, Mendes E (2010) Experiences conducting systematic reviews from novices’ perspective. In: Proceedings of the 14th international conference on evaluation and assessment in software engineering, EASE’10. http://dl.acm.org/citation.cfm?id=2227057.2227063, pp 44–53
Sackett DL (2000) Evidence-based medicine: how to practice and teach. EBM
Sjoberg DI, Dyba T, Jorgensen M (2007) The future of empirical methods in software engineering research. In: Future of software engineering, 2007. FOSE’07. IEEE, pp 358–378
Skoglund M, Runeson P (2009) Reference-based search strategies in systematic reviews. In: Proceedings of the 13th international conference on evaluation and assessment in software engineering, EASE’09, pp 31–40
Spasic I, Ananiadou S, McNaught J, Kumar A (2005) Text mining and ontologies in biomedicine: making sense of raw text. Brief Bioinform 6(3):239–251
Strauss A (1967) The discovery of grounded theory: strategies for qualitative research
Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14(3):207–222
Wohlin C (2014a) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, ACM, EASE ’14, pp 38:1–38:10
Wohlin C (2014b) Writing for synthesis of evidence in empirical software engineering. In: 8th international symposium on empirical software engineering and measurement. ACM, p 46
Wohlin C (2016) Second-generation systematic literature studies using snowballing. In: 20th International conference on evaluation and assessment in soft engineering. ACM, p 15
Zhang H, Babar MA, Bai X, Li J, Huang L (2011a) An empirical assessment of a systematic search process for systematic reviews. In: 15th international conference on evaluation and assessment in soft engineering. IET, pp 56–65
Zhang H, Babar MA, Tell P (2011b) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637
Acknowledgments
We thank the survey respondents for taking the time to complete the survey. We also acknowledge partial support from the United States National Science Foundation grant number 1305395.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Per Runeson
Rights and permissions
About this article
Cite this article
Al-Zubidy, A., Carver, J.C. Identification and prioritization of SLR search tool requirements: an SLR and a survey. Empir Software Eng 24, 139–169 (2019). https://doi.org/10.1007/s10664-018-9626-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-018-9626-5