research-article

A tertiary study: experiences of conducting systematic literature reviews in software engineering

Authors:

Mahmood NiaziAuthors Info & Claims

EASE '13: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering

Pages 177 - 182

https://doi.org/10.1145/2460999.2461025

Published: 14 April 2013 Publication History

Abstract

Context: The use of Systematic Literature Review (SLR) requires expertise and poses many challenges for novice researchers. The experiences of those who have used this research methodology can benefit novice researchers in effectively dealing with these challenges. Objective: The aim of this study is to record the reported experiences of conducting Systematic Literature Reviews, for the benefit of new researchers. Such a review will greatly benefit the researchers wanting to conduct SLR for the very first time. Method: We conducted a tertiary study to gather the experiences published by researchers. Studies that have used the SLR research methodology in software engineering and have implicitly or explicitly reported their experiences are included in this review. Results: Our research has revealed 116 studies relevant to the theme. The data has been extracted by two researchers working independently and conflicts resolved after discussion with third researcher. Findings from these studies highlight Search Strategy, Online Databases, Planning and Data Extraction as the most challenging phases of SLR. Lack of standard terminology in software engineering papers, poor quality of abstracts and problems with search engines are some of the most cited challenges. Conclusion: Further research and guidelines is required to facilitate novice researchers in conducting these phases properly.

References

[1]

Babar, M. A., and Zhang, H., 2009, Systematic Literature Reviews in Software Engineering: Preliminary Results from Interviews with Researchers, International Conference on Empirical Software Engineering and Measurement, pp. 346--355.

Digital Library

[2]

Bailey, J., Zhang, C., Budgen, D., Charters, S., and Turner, M., 2007, Search Engine Overlaps: Do they Agree or Disagree?, Second International Workshop on Realizing Evidence-Based Software Engineering, p. 2--2.

Digital Library

[3]

Beecham, S., Baddoo, N., Hall, T., Robinson, H., and Sharp, H., 2006, Protocol for a Systematic Literature Review of Motivation in Software Engineering,.

[4]

Brereton, O. P., Kitchenham, B. A., Budgen, D., Turner, M., and Khalil, M., 2007, Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain, Journal of Systems and Software, vol. 80, no. 4, pp. 571--583.

Digital Library

[5]

Cruzes, D. S and Dyba, T., 2011, Recommended Steps for Thematic Synthesis in Software Engineering, International Symposium on Empirical Software Engineering and Measurement, pp. 275--284.

Digital Library

[6]

Dieste, O., and Padua, O. A. G., 2007, Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews, Proceedings of First International Symposium on Empirical Software Engineering and Measurement, pp. 215--224.

Digital Library

[7]

Dybå, T., and Dingsøyr, T., 2008, Strength of Evidence in Systematic Reviews in Software Engineering, Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pp. 178--187.

Digital Library

[8]

Dyba, T., Dingsoyr, T., Hanssen, G. K., and SINTEF, T., 2007, Applying Systematic Reviews to Diverse Study Types: An Experience Report, First International Symposium on Empirical Software Engineering and Measurement, 2007. pp. 225--234.

Digital Library

[9]

Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., and Linkman, S., 2009, Systematic Literature Reviews in Software Engineering-A Systematic Literature Review, Information and Software Technology, vol. 51, no. 1, pp. 7--15.

Digital Library

[10]

Kitchenham, B. A., Budgen, D. and Brereton, O. P., 2006, Evidence-based Software Engineering and Systematic Literature Reviews, Product-Focused Software Process Improvement, p. 3--3, DOI=10.1007/11767718_3.

Digital Library

[11]

Kitchenham, B., and Charters, S., 2007, Guidelines for performing Systematic Literature Reviews in Software Engineering', Technical Report EBSE 2007--001, Keele University and Durham University Joint Report.

[12]

Kitchenham, B. A., Dyba, T., and Jorgensen, M., 2004, Evidence-Based Software Engineering, Proceedings of 26th International Conference on Software Engineering, pp. 273--281.

Digital Library

[13]

Kitchenham, B., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M., and Linkman, S., 2010, Systematic Literature Reviews in Software Engineering--A Tertiary Study, Information and Software Technology, vol. 52, no. 8, pp. 792--805.

Digital Library

[14]

Riaz, M., Sulayman, M., Salleh, N., and Mendes, E., 2010, Experiences Conducting Systematic Reviews from Novices' Perspective, Proceedings of 14th International Conference on Evaluation and Assessment in Software Engineering (EASE).

Digital Library

[15]

Staples, M. and Niazi, M., 2007, Experiences using systematic review guidelines, Journal of Systems and Software, vol. 80, no. 9, pp. 1425--1437.

Digital Library

[16]

Silva, F. Q. B., Santos, A. L. M., Soares, S., França, A. C. C., Monteiro, C. V. F., and Maciel, F. F., 2011, Six years of Systematic Literature Reviews in Software Engineering: An Updated Tertiary Study, Information and Software Technology, vol. 53, no. 9, pp. 899--913.

Digital Library

[17]

MacDonell, S., Shepperd, M., Kitchenham B. A., and Mendes, E., How reliable are systematic reviews in empirical software engineering?, IEEE Transactions on Software Engineering, 2010, vol. 30, no. 5, pp. 676--687.

Digital Library

Cited By

dos Santos VIwazaki AFelizardo Kde Souza ÉNakagawa E(2024)Sustainable systematic literature reviewsInformation and Software Technology10.1016/j.infsof.2024.107551(107551)Online publication date: Aug-2024
https://doi.org/10.1016/j.infsof.2024.107551
Wohlin CRuneson PHöst MOhlsson MRegnell BWesslén AWohlin CRuneson PHöst MOhlsson MRegnell BWesslén A(2024)Systematic Literature StudiesExperimentation in Software Engineering10.1007/978-3-662-69306-3_4(51-63)Online publication date: 8-May-2024
https://doi.org/10.1007/978-3-662-69306-3_4
Qayum AZhang MColreavy SChochlov MBuckley JLin DSai A(2024)A Framework and Taxonomy for Characterizing the Applicability of Software Architecture Recovery Approaches: A Tertiary‐Mapping StudySoftware: Practice and Experience10.1002/spe.336455:1(100-132)Online publication date: 17-Jul-2024
https://doi.org/10.1002/spe.3364
Show More Cited By

Index Terms

A tertiary study: experiences of conducting systematic literature reviews in software engineering
1. General and reference
  1. Document types
    1. Computing standards, RFCs and guidelines
2. Software and its engineering

Recommendations

Experience-based guidelines for effective and efficient data extraction in systematic reviews in software engineering
EASE '17: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering

To systematically collect evidence and to structure a given area in software engineering (SE), Systematic Literature Reviews (SLR) and Systematic Mapping (SM) studies have become common. Data extraction is one of the main phases (activities) when ...
The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey ...
Systematic literature reviews in software engineering - A tertiary study

Context: In a previous study, we reported on a systematic literature review (SLR), based on a manual search of 13 journals and conferences undertaken in the period 1st January 2004 to 30th June 2007. Objective: The aim of this on-going research is to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '13: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering

April 2013

268 pages

ISBN:9781450318488

DOI:10.1145/2460999

General Chair:
Fabio Q. B. da Silva
Universidade Federal de Pernambuco, Brazil
,
Program Chairs:
Natalia Juristo
Universidad Politécnica de Madrid, Spain
,
Guilherme Horta Travassos
Universidade Federal do Rio de Janeiro, Brazil

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Centro de Informatica - UFPE: Centro de Informatica - UFPE
SBC: Brazilian Computer Society
CNPq: Conselho Nacional de Desenvolvimento Cientifico e Tecn
CAPES: Brazilian Higher Education Funding Council

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

EASE '13

Sponsor:

Centro de Informatica - UFPE
SBC
CNPq
CAPES

EASE '13: 17th International Conference on Evaluation and Assessment in Software Engineering

April 14 - 16, 2013

Porto de Galinhas, Brazil

Acceptance Rates

EASE '13 Paper Acceptance Rate 31 of 94 submissions, 33%;

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
951
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)9

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

dos Santos VIwazaki AFelizardo Kde Souza ÉNakagawa E(2024)Sustainable systematic literature reviewsInformation and Software Technology10.1016/j.infsof.2024.107551(107551)Online publication date: Aug-2024
https://doi.org/10.1016/j.infsof.2024.107551
Wohlin CRuneson PHöst MOhlsson MRegnell BWesslén AWohlin CRuneson PHöst MOhlsson MRegnell BWesslén A(2024)Systematic Literature StudiesExperimentation in Software Engineering10.1007/978-3-662-69306-3_4(51-63)Online publication date: 8-May-2024
https://doi.org/10.1007/978-3-662-69306-3_4
Qayum AZhang MColreavy SChochlov MBuckley JLin DSai A(2024)A Framework and Taxonomy for Characterizing the Applicability of Software Architecture Recovery Approaches: A Tertiary‐Mapping StudySoftware: Practice and Experience10.1002/spe.336455:1(100-132)Online publication date: 17-Jul-2024
https://doi.org/10.1002/spe.3364
Kotti ZGalanopoulou RSpinellis D(2023)Machine Learning for Software Engineering: A Tertiary StudyACM Computing Surveys10.1145/357290555:12(1-39)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3572905
Li ZRainer A(2023)Reproducible Searches in Systematic Reviews: An Evaluation and GuidelinesIEEE Access10.1109/ACCESS.2023.329921111(84048-84060)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3299211
Badami MBenatallah BBaez M(2023)Adaptive search query generation and refinement in systematic literature reviewInformation Systems10.1016/j.is.2023.102231117:COnline publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.is.2023.102231
Li ZRainer AKiss ÁMarín BSaadatmand M(2022)Academic search engines: constraints, bugs, and recommendationsProceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation10.1145/3548659.3561310(25-32)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3548659.3561310
Iwazaki Ados Santos VFelizardo Kde Souza EValentim NNakagawa E(2022)Benefits and Challenges of a Graduate Course: An Experience Teaching Systematic Literature Review2022 IEEE Frontiers in Education Conference (FIE)10.1109/FIE56618.2022.9962446(1-9)Online publication date: 8-Oct-2022
https://doi.org/10.1109/FIE56618.2022.9962446
Alves LVasconcellos FNogueira B(2022)SeSG: a search string generator for Secondary Studies with hybrid search strategies using text miningEmpirical Software Engineering10.1007/s10664-021-10084-427:5Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1007/s10664-021-10084-4
Felizardo KRamos AO. Melo Cde Souza ÉVijaykumar NNakagawa E(2021)Global and Latin American female participation in evidence-based software engineering: a systematic mapping studyJournal of the Brazilian Computer Society10.1186/s13173-021-00109-727:1Online publication date: 19-Mar-2021
https://doi.org/10.1186/s13173-021-00109-7
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten