Skip to main content
Springer Nature Link
Log in

Identification and prioritization of SLR search tool requirements: an SLR and a survey

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Context

The number of published systematic literature reviews (SLRs) in software engineering venues is increasing. However, even with their high adaptation rate, the task of performing an SLR requires a large amount of effort and presents a number of barriers. Specifically, during the SLR search phase authors must expend a lot of time and overcome a large number of barriers.

Objective

To help alleviate some of the barriers in the search phase, we identify and prioritize SLR search tool requirements based on input from the SLR community. These requirements will help tool builders ensure they focus their efforts appropriately.

Method

We conducted an SLR and a survey of SLR authors in software engineering. In the SLR we extracted problems and solutions SLR authors reported during their search processes. In the survey we asked respondents to describe the problems they faced during SLR search and to specify any requirements they would like to see as part of an SLR search tool. We also asked survey respondents to describe any tools they use to support SLR search, along with the strengths and limitations of those tools.

Results

Based on analysis of 84 studies from the literature and a qualitative analysis of 131 responses from researchers, we identified a set of functional requirements that authors, researchers, and tool builders can use as a reference. We also describe the tools currently used by SLR researchers.

Conclusions

The list of SLR search tool requirements can be used by tool builders as a guide when constructing new tools. Our analysis of tools showed that more recent tools are covering more of the requirements than in the past.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. http://carver.cs.ua.edu/Data/2018/SLR_Search/Papers.htm

  2. http://carver.cs.ua.edu/Data/2018/SLR_Search/ProblemSources.htm

  3. http://carver.cs.ua.edu/Data/2018/SLR_Search/SLR-Tools/

  4. http://carver.cs.ua.edu/Data/2018/SLR_Search/Tools_Analysis.xlsx

  5. http://carver.cs.ua.edu/Data/2018/SLR_Search/SLR-Tools/

References

  • Al-Zubidy A, Carver JC, Hale DP, Hassler EE (2017) Vision for {SLR} tooling infrastructure: prioritizing value-added requirements. Inf Softw Technol. https://doi.org/10.1016/j.infsof.2017.06.007

  • Allman-Farinelli M, Byron A, Collins C, Gifford J, Williams P (2014) Challenges and lessons from systematic literature reviews for the australian dietary guidelines. Aust J Prim Health 20(3):236–240

    Article  Google Scholar 

  • Babar MA, Zhang H (2009) Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In: 3rd international symposium on empirical software engineering and measurement. IEEE Computer Society, pp 346–355

  • Badampudi D, Wohlin C, Petersen K (2015) Experiences from using snowballing and database searches in systematic literature studies. In: 19th international conference on evaluation and assessment in software engineering. ACM, p 17

  • Barat S, Clark T, Barn B, Kulkarni V (2017) A model-based approach to systematic review of research literature. In: Proceedings of the 10th innovations in software engineering conference. ACM, pp 15–25

  • Barn B, Raimondi F, Athappian L, Clark T (2014) Slrtool: a tool to support collaborative systematic literature reviews. In: ICEIS 2014—Proceedings of the 16th international conference on enterprise information systems, vol 2, pp 440–447

  • Biolchini J, Mian PG, Natali ACC, Travassos GH (2005) Systematic review in software engineering. System Engineering and Computer Science Department COPPE/UFRJ. Technical Report ES 679(05):45

  • Bowes D, Hall T, Beecham S (2012) Slurp: a tool to help large complex systematic literature reviews deliver valid and rigorous results. In: Proceedings of the 2nd international workshop on evidential assessment of software technologies. ACM, pp 33–36

  • Brereton P, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583

    Article  Google Scholar 

  • Budgen D, Brereton P, Drummond S, Williams N (2018) Reporting systematic reviews: some lessons from a tertiary study. Inf Softw Technol 95:62–74. https://doi.org/10.1016/j.infsof.2017.10.017

    Article  Google Scholar 

  • Carver JC, Hassler E, Hernandes E, Kraft NA (2013) Identifying barriers to the systematic literature review process. In: 7th international symposium on empirical software engineering and measurement. IEEE, pp 203–212

  • Corbin J, Strauss A et al (2008) Basics of qualitative research: techniques and procedures for developing grounded theory. Thousand Oaks

  • de Almeida Biolchini JC, Mian PG, Natali ACC, Conte TU, Travassos GH (2007) Scientific research ontology to support systematic review in software engineering. Adv Eng Inf 21(2):133–151

    Article  Google Scholar 

  • Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. Empir Softw Eng 14(5):513–539

    Article  Google Scholar 

  • Dyba T, Dingsoyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: an experience report. In: 1st international symposium on empirical software engineering and measurement. IEEE, pp 225–234

  • Engström E, Skoglund M, Runeson P (2008) Empirical evaluations of regression test selection techniques: a systematic review. In: 1st international symposium on empirical software engineering and measurement. ACM, pp 22–31

  • Felizardo KR, Mendes E, Kalinowski M, Souza ÉF, Vijaykumar NL (2016) Using forward snowballing to update systematic reviews in soft. engineering. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement. ACM, p 53

  • Fernández-Sáez A M, Bocco MG, Romero FP (2010) Slr-tool: a tool for performing systematic literature reviews. In: ICSOFT (2), pp 157–166

  • Hassler E, Carver JC, Kraft NA, Hale D (2014) Outcomes of a community workshop to identify and rank barriers to the systematic literature review process. In: 18th International conference on evaluation & assessment in software engineering. ACM, p 31

  • Hassler E, Carver JC, Hale D, Al-Zubidy A (2016) Identification of SLR tool needs–results of a community workshop. J Inf Softw Technol 70:122–129

    Article  Google Scholar 

  • Hernandes E, Zamboni A, Fabbri S, Thommazo AD (2012) Using gqm and tam to evaluate start-a tool that supports systematic review. CLEI Electron J 15 (1):3–3

    Article  Google Scholar 

  • Jalali S, Wohlin C (2012) Systematic literature studies: database searches vs. backward snowballing. In: Proceedings of the ACM-IEEE international symposium on empirical software engineering and measurement. ACM, pp 29–38

  • Kitchenham B (2004) Procedures for performing systematic reviews. Tech. Rep. TR/SE-0401, Keel University

  • Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075

    Article  Google Scholar 

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE-2007-01, Keel University

  • Kitchenham B, Linkman S, Law D (1997) Desmet: a methodology for evaluating software engineering methods and tools. Comput Control Eng J 8(3):120–126

    Article  Google Scholar 

  • Kitchenham BA, Dyba T, Jorgensen M (2004) Evidence-based software engineering. In: 26th international conference on software engineering. IEEE Computer Society, pp 273–281

  • Kitchenham B, Pretorius R, Budgen D, Brereton OP, Turner M, Niazi M, Linkman S (2010) Systematic literature reviews in software engineering–a tertiary study. Inf Softw Technol 52(8):792– 805

    Article  Google Scholar 

  • Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews, vol 4. CRC Press, Boca Raton

    Google Scholar 

  • Mallett R, Hagen-Zanker J, Slater R, Duvendack M (2012) The benefits and challenges of using systematic reviews in international development research. J Dev Effect 4(3):445–455

    Article  Google Scholar 

  • Marshall C, Brereton P (2013) Tools to support systematic literature reviews in software engineering: a mapping study. In: 7th international symposium on empirical software engineering and measurement. IEEE, pp 296–299

  • Marshall C, Brereton P, Kitchenham B (2014) Tools to support systematic reviews in software engineering: a feature analysis. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. ACM, p 13

  • Molléri JS, Benitti FBV (2015) Sesra: a web-based automated tool to support the systematic literature review process. In: Proceedings of the 19th international conference on evaluation and assessment in software engineering. ACM, p 24

  • Parsifal (2015) Parsifal systematic literature review tool. [Online; accessed 1–December-2016]

  • Perestelo-Pérez L (2013) Standards on how to develop and report systematic reviews in psychology and health. Int J Clin Health Psychol 13(1):49–57. https://doi.org/10.1016/S1697-2600(13)70007-3

    Article  Google Scholar 

  • Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: EASE, vol 8, pp 68–77

  • Pignone M, Saha S, Hoerger T, Lohr KN, Teutsch S, Mandelblatt J (2005) Challenges in systematic reviews of economic analyses. Ann Intern Med 142 (12_Part_2):1073–1079

    Article  Google Scholar 

  • Reed D, Price EG, Windish DM, Wright SM, Gozu A, Hsu EB, Beach MC, Kern D, Bass EB (2005) Challenges in systematic reviews of educational intervention studies. Ann Intern Med 142(12_Part_2):1080–1089

    Article  Google Scholar 

  • Riaz M, Sulayman M, Salleh N, Mendes E (2010) Experiences conducting systematic reviews from novices’ perspective. In: Proceedings of the 14th international conference on evaluation and assessment in software engineering, EASE’10. http://dl.acm.org/citation.cfm?id=2227057.2227063, pp 44–53

  • Sackett DL (2000) Evidence-based medicine: how to practice and teach. EBM

  • Sjoberg DI, Dyba T, Jorgensen M (2007) The future of empirical methods in software engineering research. In: Future of software engineering, 2007. FOSE’07. IEEE, pp 358–378

  • Skoglund M, Runeson P (2009) Reference-based search strategies in systematic reviews. In: Proceedings of the 13th international conference on evaluation and assessment in software engineering, EASE’09, pp 31–40

  • Spasic I, Ananiadou S, McNaught J, Kumar A (2005) Text mining and ontologies in biomedicine: making sense of raw text. Brief Bioinform 6(3):239–251

    Article  Google Scholar 

  • Strauss A (1967) The discovery of grounded theory: strategies for qualitative research

  • Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14(3):207–222

    Article  Google Scholar 

  • Wohlin C (2014a) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, ACM, EASE ’14, pp 38:1–38:10

  • Wohlin C (2014b) Writing for synthesis of evidence in empirical software engineering. In: 8th international symposium on empirical software engineering and measurement. ACM, p 46

  • Wohlin C (2016) Second-generation systematic literature studies using snowballing. In: 20th International conference on evaluation and assessment in soft engineering. ACM, p 15

  • Zhang H, Babar MA, Bai X, Li J, Huang L (2011a) An empirical assessment of a systematic search process for systematic reviews. In: 15th international conference on evaluation and assessment in soft engineering. IET, pp 56–65

  • Zhang H, Babar MA, Tell P (2011b) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637

    Article  Google Scholar 

Download references

Acknowledgments

We thank the survey respondents for taking the time to complete the survey. We also acknowledge partial support from the United States National Science Foundation grant number 1305395.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Alabama, Tuscaloosa, AL, USA

    Ahmed Al-Zubidy & Jeffrey C. Carver

Authors
  1. Ahmed Al-Zubidy
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jeffrey C. Carver
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Jeffrey C. Carver.

Additional information

Communicated by: Per Runeson

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Zubidy, A., Carver, J.C. Identification and prioritization of SLR search tool requirements: an SLR and a survey. Empir Software Eng 24, 139–169 (2019). https://doi.org/10.1007/s10664-018-9626-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9626-5

Keywords