skip to main content
10.1145/2601248.2601274acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
research-article

Outcomes of a community workshop to identify and rank barriers to the systematic literature review process

Published: 13 May 2014 Publication History

Abstract

Systematic Literature Reviews (SLRs) are an important tool used by software engineering researchers to summarize the state of knowledge about a particular topic. Currently, SLR authors must perform the difficult, time-consuming task in largely manual fashion. To identify barriers faced by SLR authors, we conducted an interactive community workshop prior to ESEM'13. Workshop participants generated a total of 100 ideas that, through group discussions, formed 37 composite barriers to the SLR process. Further analysis reveals the barriers relate to latent themes regarding the SLR process, primary studies, the practitioner community, and tooling. This paper describes the barriers identified during the workshop along with a ranking of those barriers that is based on votes by workshop attendees. The paper concludes by describing the impact of these barriers on three important constituencies: SLR Methodology Researchers, SLR Authors and SLR consumers.

References

[1]
D. Bowes, T. Hall, and S. Beecham. Slurp: A tool to help large complex systematic literature reviews deliver valid and rigorous results. In Proceedings of the 2nd International Workshop on Evidential Assessment of Software Technologies, EAST '12, pages 33--36, New York, NY, USA, 2012. ACM.
[2]
J. C. Carver, E. Hassler, E. Hernandes, and N. A. Kraft. Identifying barriers to the systematic literature review process. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pages 203--212, 2013.
[3]
D. Cruzes, V. R. Basili, F. Shull, and M. Jino. Automated information extraction from empirical software engineering literature: Is that possible? In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement, pages 491--493, 2007.
[4]
D. Cruzes, M. Mendonca, V. Basili, F. Shull, and M. Jino. Using context distance measurement to analyze results across studies. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement, pages 235--244, 2007.
[5]
A. Delbecq, A. Van de Ven, and D. Gustafson. Group techniques for program planning: a guide to nominal group and Delphi processes. Management applications series. Scott, Foresman, 1975.
[6]
K. Felizardo, N. Salleh, R. Martins, E. Mendes, S. MacDonell, and J. Maldonado. Using visual text mining to support the study selection activity in systematic literature reviews. In Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement, pages 77--86, 2011.
[7]
K. R. Felizardo, G. F. Andery, F. V. Paulovich, R. Minghim, and J. C. Maldonado. A visual analysis approach to validate the selection review of primary studies in systematic reviews. Information and Software Technology, 54(10):1079--1091, 2012.
[8]
K. R. Felizardo, E. Y. Nakagawa, D. Feitosa, R. Minghim, and J. C. Maldonado. An approach based on visual text mining to support categorization and classification in the systematic mapping. In Proceedings of the 14th International Conference on Evaluation and Assessment in Software Engineering, EASE'10, pages 34--43, Swinton, UK, UK, 2010. British Computer Society.
[9]
K. R. Felizardo, M. Riaz, M. Sulayman, E. Mendes, S. G. MacDonell, and J. C. Maldonado. Analysing the use of graphs to represent the results of systematic reviews in software engineering. In Proceedings of the 25th Brazilian Symposium on Software Engineering, SBES '11, pages 174--183, Washington, DC, USA, 2011. IEEE Computer Society.
[10]
A. M. Fernández-Sáez, M. G. Bocco, and F. P. Romero. Slr-tool --- a tool for performing systematic literature reviews. In ICSOFT (2), pages 157--166, 2010.
[11]
M. Ghafari, M. Saleh, and T. Ebrahimi. A federated search approach to facilitate systematic literature review in software engineering. International Journal of Software Engineering and Applications, 3(2):13--24, 2012.
[12]
E. Hernandes, A. Zamboni, S. Fabbri, and A. Di Thommazo. Using GQM and TAM to evaluate StArt-a tool that supports systematic review. CLEI Electronic Journal, 15(1):13--25, 2012.
[13]
A. Kakar and J. Carver. Best practices for managing the fuzzy front-end of software development (sd): Insights from a systematic review of new product development (npd) literature. In Proceedings of International Research Workshop on IT Project Management, 2012.
[14]
B. Kitchenham. Procedures for performing systematic reviews. Joint Technical Report TR/SE-0401 and NICTA 0400011T.1, Keele University, Keele, UK, 2004.
[15]
B. Kitchenham and P. Brereton. A systematic review of systematic review process research in software engineering. Information and Software Technology, 55(12):2049--2075, 2013.
[16]
B. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report EBSE-2007-01, Keel University, 2007.
[17]
T. C. Lethbridge, S. E. Sim, and J. Singer. Studying software engineers: Data collection techniques for software field studies. Empirical Software Engineering, 5(3):311--341, 2000.
[18]
V. Malheiros, E. Hohn, R. Pinho, and M. Mendonca. A visual text mining approach for systematic reviews. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement, pages 245--254, 2007.
[19]
C. Marshall and P. Brereton. Tools to support systematic literature reviews in software engineering: A mapping study. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pages 296--299, 2013.
[20]
J. Pate, R. Tairas, and N. Kraft. Clone evolution: a systematic review. Journal of Software: Evolution and Process, 25(3):261--283, 2013.
[21]
A. Strauss and J. Corbin. Basics of qualitative research: Grounded theory procedures and techniques, volume 2nd. Sage, 1990.
[22]
Y. Sun, Y. Yang, H. Zhang, W. Zhang, and Q. Wang. Towards evidence-based ontology for supporting systematic literature review. In Proceedings of the 16th International Conference on Evaluation Assessment in Software Engineering, pages 171--175, 2012.
[23]
F. Tomassetti, G. Rizzo, A. Vetro, L. Ardito, M. Torchiano, and M. Morisio. Linked data approach for selection process automation in systematic reviews. In Proceedings of the 15th International Conference on Evaluation Assessment in Software Engineering, pages 31--35, 2011.
[24]
J. A. S. Torres, D. S. Cruzes, and L. do Nascimento Salvador. Automatic results identification in software engineering papers. is it possible? In Proceedings of the 12th International Conference on Computational Science and Its Applications, pages 108--112, 2012.
[25]
G. S. Walia and J. C. Carver. A systematic literature review to identify and classify software requirement errors. Information and Software Technology, 51(7):1087--1109, July 2009.
[26]
B. J. Williams and J. C. Carver. Characterizing software architecture changes: A systematic review. Information and Software Technology, 52(1):31--51, Jan. 2010.

Cited By

View all
  • (2024)Nudge Evidence Briefing: A Proposal for Transferring Scientific Knowledge about NudgesProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702073(1-11)Online publication date: 7-Oct-2024
  • (2024)Sustainable systematic literature reviewsInformation and Software Technology10.1016/j.infsof.2024.107551(107551)Online publication date: Aug-2024
  • (2024)On the application of image augmentation for plant disease detection: A systematic literature reviewSmart Agricultural Technology10.1016/j.atech.2024.1005909(100590)Online publication date: Dec-2024
  • Show More Cited By

Index Terms

  1. Outcomes of a community workshop to identify and rank barriers to the systematic literature review process

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
    May 2014
    486 pages
    ISBN:9781450324762
    DOI:10.1145/2601248
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    • Brunel University: Brunel University

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 13 May 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. empirical software engineering
    2. systematic literature review
    3. workshop

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    EASE '14
    Sponsor:
    • Brunel University

    Acceptance Rates

    Overall Acceptance Rate 71 of 232 submissions, 31%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)28
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Nudge Evidence Briefing: A Proposal for Transferring Scientific Knowledge about NudgesProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702073(1-11)Online publication date: 7-Oct-2024
    • (2024)Sustainable systematic literature reviewsInformation and Software Technology10.1016/j.infsof.2024.107551(107551)Online publication date: Aug-2024
    • (2024)On the application of image augmentation for plant disease detection: A systematic literature reviewSmart Agricultural Technology10.1016/j.atech.2024.1005909(100590)Online publication date: Dec-2024
    • (2024)Using rapid reviews to support software engineering practice: a systematic review and a replication studyEmpirical Software Engineering10.1007/s10664-024-10545-630:1Online publication date: 25-Oct-2024
    • (2024)Artificial intelligence for literature reviews: opportunities and challengesArtificial Intelligence Review10.1007/s10462-024-10902-357:10Online publication date: 17-Aug-2024
    • (2024)Comparing the Efficacy of Rapid Review With a Systematic Review in the Software Engineering FieldJournal of Software: Evolution and Process10.1002/smr.2748Online publication date: 3-Dec-2024
    • (2023)Roadmap to Reasoning in Microservice Systems: A Rapid ReviewApplied Sciences10.3390/app1303183813:3(1838)Online publication date: 31-Jan-2023
    • (2023)Evidence Profiles for Validity Threats in Program Comprehension ExperimentsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00162(1907-1919)Online publication date: 14-May-2023
    • (2023)On the use of deep learning in software defect predictionJournal of Systems and Software10.1016/j.jss.2022.111537195(111537)Online publication date: Jan-2023
    • (2022)A longitudinal case study on the effects of an evidence-based software engineering trainingProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3510456.3514150(1-13)Online publication date: 21-May-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media