Skip to main content
Springer Nature Link
Log in

SWARM-SLR - Streamlined Workflow Automation for Machine-Actionable Systematic Literature Reviews

  • Conference paper
  • First Online:
Linking Theory and Practice of Digital Libraries (TPDL 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15177))

Included in the following conference series:

Abstract

Authoring survey or review articles still requires significant tedious manual effort, despite many advancements in research knowledge management having the potential to improve efficiency, reproducibility, and reuse. However, these advancements bring forth an increasing number of approaches, tools, and systems, which often cover only specific stages and lack a comprehensive workflow utilizing their task-specific strengths. We propose the Streamlined Workflow Automation for Machine-actionable Systematic Literature Reviews (SWARM-SLR) to crowdsource the improvement of SLR efficiency while maintaining scientific integrity in a state-of-the-art knowledge discovery and distribution process. The workflow aims to domain-independently support researchers in collaboratively and sustainably managing the rising scholarly knowledge corpus. By synthesizing guidelines from the literature, we have composed a set of 65 requirements, spanning from planning to reporting a review. Existing tools were assessed against these requirements and synthesized into the SWARM-SLR workflow prototype, a ready-for-operation software support tool. The SWARM-SLR was evaluated via two online surveys, which largely confirmed the validity of the 65 requirements and situated 11 tools to the different life-cycle stages. The SWARM-SLR workflow was similarly evaluated and found to be supporting almost the entire span of an SLR, excelling specifically in search and retrieval, information extraction, knowledge synthesis, and distribution. Our SWARM-SLR requirements and workflow support tool streamlines the SLR support for researchers, allowing sustainable collaboration by linking individual efficiency improvements to crowdsourced knowledge management. If these efforts are continued, we expect the increasing number of tools to be manageable and usable inside fully structured, (semi-)automated literature review workflows.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/borgnetzwerk/tools/tree/main/scripts/SWARM-SLR.

  2. 2.

    Recommendation system template: https://orkg.org/template/R673347/.

  3. 3.

    Recommendation systems comparison: https://orkg.org/comparison/R674188/.

  4. 4.

    Requirement review survey: https://survey.uni-hannover.de/index.php/555283.

  5. 5.

    Tool assessment survey: https://survey.uni-hannover.de/index.php/628237.

  6. 6.

    https://github.com/borgnetzwerk/tools/blob/main/scripts/SWARM-SLR/data/.

References

  1. Open Knowledge Maps: A Visual Interface to the World’s Scientific Knowledge (2019). https://openknowledgemaps.org

  2. Copyright Registration Guidance: Works Containing Material Generated by Artificial Intelligence (2023). https://www.federalregister.gov/documents/2023/03/16/2023-05321/copyright-registration-guidance-works-containing-material-generated-by-artificial-intelligence

  3. The Systematic Review Toolbox (2023). http://www.systematicreviewtools.com/guidance.php

  4. arXiv.org e-Print archive (2024). https://arxiv.org/

  5. Connected Papers | Find and explore academic papers (2024). https://www.connectedpapers.com/

  6. Adams, J., Khan, H., Raeside, R., White, D.: Research Methods for Graduate Business and Social Science Students. SAGE Publications India Pvt Ltd (2007). https://doi.org/10.4135/9788132108498

  7. Armstrong, K.: ChatGPT: US lawyer admits using AI for case research (2023). https://www.bbc.com/news/world-us-canada-65735769

  8. Beel, J., Gipp, B., Langer, S., Breitinger, C.: Research-paper recommender systems: a literature survey. Int. J. Digit. Libr. 17(4), 305–338 (2016). https://doi.org/10.1007/s00799-015-0156-0

    Article  Google Scholar 

  9. Berquand, A.: Text mining and natural language processing for the early stages of space mission design. Ph.D. thesis, University of Strathclyde (2021). https://doi.org/10.48730/95nx-rc75

  10. Bless, C., Baimuratov, I., Karras, O.: SciKGTeX - a LaTeX package to semantically annotate contributions in scientific publications. In: Proceedings of the 23nd ACM/IEEE Joint Conference on Digital Libraries. ACM (2023)

    Google Scholar 

  11. BorgNetzWerk: borgnetzwerk/tools (2024). https://github.com/borgnetzwerk/tools. Original-date: 2023-01-24T14:28:59Z

  12. Borisova, E., Ahmad, R.A., Rehm, G.: Open Science Best Practices in Data Science and Artificial Intelligence, vol. 1 (2023). https://doi.org/10.52825/cordi.v1i.299

  13. Bornmann, L., Mutz, R.: Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J. Am. Soc. Inf. Sci. 66(11), 2215–2222 (2015). https://doi.org/10.1002/asi.23329

    Article  Google Scholar 

  14. Bosman, J., Kramer, B.: Innovations in scholarly communication - data of the global 2015-2016 survey (2016). https://doi.org/10.5281/zenodo.49583

  15. Bosman, J., Kramer, B.: 400+ tools and innovations in scholarly communication - data collection forms (2023). https://docs.google.com/spreadsheets/d/1KUMSeq_Pzp4KveZ7pb5rddcssk1XBTiLHniD0d3nDqo/edit

  16. Bosman, J., Kramer, B.: Tools that love to be together - template per tool (2023). https://docs.google.com/spreadsheets/d/1d2YSAmYGEw1WTMHk2Wfz-L155bjxT6eQkO772FJko8E/edit

  17. Clark, J., Glasziou, P., Mar, C.D., Bannach-Brown, A., Stehlik, P., Scott, A.M.: A full systematic review was completed in 2 weeks using automation tools: a case study. J. Clin. Epidemiol. 121, 81–90 (2020). https://doi.org/10.1016/j.jclinepi.2020.01.008

    Article  Google Scholar 

  18. Gorashy, Z., Salim, N.: Systematic literature review (SLR) automation: a systematic literature review. J. Theor. Appl. Inf. Technol. 59, 661–672 (2014)

    Google Scholar 

  19. Higgins, J., et al.: Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated august 2023) (2023). www.training.cochrane.org/handbook

  20. Hussein, H., Farfar, K.E., Oelen, A., Karras, O., Auer, S.: Increasing reproducibility in science by interlinking semantic artifact descriptions in a knowledge graph. In: Goh, D.H., Chen, S.J., Tuarob, S. (eds.) ICADL 2023, Part II. LNCS, vol. 14458, pp. 220–229. Springer, Heidelberg (2023). https://doi.org/10.1007/978-981-99-8088-8_19

    Chapter  Google Scholar 

  21. Ioannidis, J.P.A.: The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 94 3, 485–514 (2016). https://api.semanticscholar.org/CorpusID:25375827

  22. Jaradeh, M.Y., Auer, S., Prinz, M., Kovtun, V., Kismihók, G., Stocker, M.: Open Research Knowledge Graph: Towards Machine Actionability in Scholarly Communication (2019)

    Google Scholar 

  23. Karras, O., Wernlein, F., Klünder, J., Auer, S.: Divide and conquer the EmpiRE: a community-maintainable knowledge graph of empirical research in requirements engineering. In: 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1–12. IEEE (2023)

    Google Scholar 

  24. Kitchenham, B.A., et al.: Refining the systematic literature review process-two participant-observer case studies. Empirical Softw. Eng. 15(6), 618–653 (2010). https://doi.org/10.1007/s10664-010-9134-8

    Article  Google Scholar 

  25. Kohl, C., et al.: Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environ. Evid. 7(1), 8 (2018). https://doi.org/10.1186/s13750-018-0115-5

    Article  Google Scholar 

  26. Kraker, P., Kittel, C., Enkhbayar, A.: Open knowledge maps: creating a visual interface to the world’s scientific knowledge based on natural language processing. 027.7 Zeitschrift für Bibliothekskultur 4(2), 98–103 (2016). https://doi.org/10.5281/zenodo.4705327

  27. Kreutz, C.K., Schenkel, R.: Scientific paper recommendation systems: a literature review of recent publications. Int. J. Digit. Libr. 23(4), 335–369 (2022). https://doi.org/10.1007/s00799-022-00339-w

    Article  Google Scholar 

  28. Larsen, K., Hovorka, D., Dennis, A., West, J.: Understanding the Elephant: The Discourse Approach to Boundary Identification and Corpus Construction for Theory Review Articles (2018). https://doi.org/10.17705/1jais.00556

  29. Liu, C., Ali, N.L.: Co-citation and bibliographic coupling based on connected papers: review of public opinion research in a broad sense in the west. Asian Soc. Sci. 18(7), 29 (2022). https://doi.org/10.5539/ass.v18n7p29

    Article  Google Scholar 

  30. Liu, Y., et al.: Summary of ChatGPT/GPT-4 research and perspective towards the future of large language models (2023). https://doi.org/10.48550/arXiv.2304.01852

  31. Machi, L., McEvoy, B.: The Literature Review: Six Steps to Success. SAGE Publications (2012). https://books.google.de/books?id=QJ8nmTc4mnAC

  32. Marshall, I.J., Wallace, B.C.: Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systems Control Found. Appl. 8(1), 163 (2019). https://doi.org/10.1186/s13643-019-1074-9

    Article  Google Scholar 

  33. Martín-Martín, A., Orduna-Malea, E., Thelwall, M., López-Cózar, E.D.: Google scholar, web of science, and scopus: a systematic comparison of citations in 252 subject categories. J. Informetrics 12(4), 1160–1177 (2018). https://doi.org/10.1016/j.joi.2018.09.002

  34. Molleri, J., Silva, L., Benitti, F.: Proposal of an automated approach to support the systematic review of literature process (2013)

    Google Scholar 

  35. Niaksu, O., Skinulyte, J., Duhaze, H.G.: A systematic literature review of data mining applications in healthcare. In: Huang, Z., Liu, C., He, J., Huang, G. (eds.) WISE 2013. LNCS, vol. 8182, pp. 313–324. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54370-8_27

    Chapter  Google Scholar 

  36. Novelli, C., Casolari, F., Rotolo, A., Taddeo, M., Floridi, L.: Taking AI risks seriously: a new assessment model for the AI act (2023). https://doi.org/10.2139/ssrn.4447964

  37. Olorisade, B.K., Brereton, P., Andras, P.: Reproducibility of studies on text mining for citation screening in systematic reviews: evaluation and checklist. J. Biomed. Inform. 73, 1–13 (2017). https://doi.org/10.1016/j.jbi.2017.07.010

    Article  Google Scholar 

  38. Pulsiri, N., Vatananan-Thesenvitz, R.: Improving systematic literature review with automation and bibliometrics. In: 2018 Portland International Conference on Management of Engineering and Technology (PICMET), pp. 1–8 (2018). https://doi.org/10.23919/PICMET.2018.8481746. ISSN: 2159-5100

  39. Ralph, P., Baltes, S.: Paving the way for mature secondary research: the seven types of literature review. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, pp. 1632–1636. Association for Computing Machinery (2022). https://doi.org/10.1145/3540250.3560877

  40. Robson, C., McCartan, K.: Real World Research, 4 edn. Wiley (2016). https://www.perlego.com/book/1485149/real-world-research-pdf

  41. Ros, R., Bjarnason, E., Runeson, P.: A machine learning approach for semi-automated search and selection in literature studies. In: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE 2017, pp. 118–127. Association for Computing Machinery (2017). https://doi.org/10.1145/3084226.3084243

  42. Rupp, C., die, S.: Requirements-engineering und -management. In: Requirements-Engineering und -Management, pp. I–6. Carl Hanser Verlag GmbH & Co. KG (2014). https://doi.org/10.3139/9783446443136.fm

  43. Shabanov, I.: The Effortless Literature Review (2023). https://ilyashabanov.substack.com/p/the-effortless-literature-review

  44. Stocker, M., et al.: FAIR scientific information with the open research knowledge graph. FAIR Connect 1(1), 19–21 (2023). https://doi.org/10.3233/FC-221513

    Article  Google Scholar 

  45. Templier, M., Paré, G.: Transparency in literature reviews: an assessment of reporting practices across review types and genres in top IS journals. Eur. J. Inf. Syst. 27(5), 503–550 (2018). https://doi.org/10.1080/0960085X.2017.1398880

    Article  Google Scholar 

  46. Thomas, J., et al.: Living systematic review network: living systematic reviews: 2. Combining Hum. Mach. Effort 91, 31–37 (2017). https://doi.org/10.1016/j.jclinepi.2017.08.011

    Article  Google Scholar 

  47. Tomassetti, F., Rizzo, G., Vetro, A., Ardito, L., Torchiano, M., Morisio, M.: Linked data approach for selection process automation in systematic reviews. In: 15th Annual Conference on Evaluation & Assessment in Software Engineering (EASE 2011), pp. 31–35 (2011). https://doi.org/10.1049/ic.2011.0004

  48. Torraco, R.J.: Writing integrative literature reviews: using the past and present to explore the future. Hum. Resour. Dev. Rev. 15(4), 404–428 (2016). https://doi.org/10.1177/1534484316671606

    Article  Google Scholar 

  49. Tsafnat, G., Dunn, A., Glasziou, P., Coiera, E.: The automation of systematic reviews. BMJ 346, f139 (2013). https://doi.org/10.1136/bmj.f139

  50. Tsafnat, G., Glasziou, P., Choong, M.K., Dunn, A., Galgani, F., Coiera, E.: Systematic review automation technologies. Syst. Rev. 3(1), 74 (2014). https://doi.org/10.1186/2046-4053-3-74

    Article  Google Scholar 

  51. Wagner, G., Lukyanenko, R., Paré, G.: Artificial intelligence and the conduct of literature reviews. J. Inf. Technol. 37(2), 209–226 (2022). https://doi.org/10.1177/02683962211048201

    Article  Google Scholar 

  52. Wallace, B.C., Dahabreh, I.J., Schmid, C.H., Lau, J., Trikalinos, T.A.: Modernizing the systematic review process to inform comparative effectiveness: tools and methods. J. Comp. Effectiveness Res. 2(3), 273–282 (2013). https://doi.org/10.2217/cer.13.17

    Article  Google Scholar 

  53. Wallace, B.C., et al.: Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genetics Med. 14(7), 663–669 (2012). https://doi.org/10.1038/gim.2012.7

    Article  Google Scholar 

  54. Wallace, B.C., Trikalinos, T.A., Lau, J., Brodley, C., Schmid, C.H.: Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinform. 11(1), 55 (2010). https://doi.org/10.1186/1471-2105-11-55

    Article  Google Scholar 

  55. Wilkinson, M.D., et al.: The FAIR guiding principles for scientific data management and stewardship. Sci. Data 3(1), 160018 (2016). https://doi.org/10.1038/sdata.2016.18

    Article  Google Scholar 

  56. Xiao, D.: Research Guides: AI-Based Literature Review Tools: Home (2023). https://tamu.libguides.com/c.php?g=1289555&p=9470549

Download references

Acknowledgements

This work was co-funded by the DFG SE2A Excellence Cluster, as well as the NFDI4Ing project funded by the German Research Foundation (project number 442146713) and NFDI4DataScience (project number 460234259).

Author information

Authors and Affiliations

  1. L3S Research Center, Leibniz University Hannover, Hannover, Germany

    Tim Wittenborg & Sören Auer

  2. Cluster of Excellence SE2A - Sustainable and Energy-Efficient Aviation, Technische Universität Braunschweig, Braunschweig, Germany

    Tim Wittenborg & Sören Auer

  3. TIB - Leibniz Information Centre for Science and Technology, Hannover, Germany

    Oliver Karras & Sören Auer

Authors
  1. Tim Wittenborg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oliver Karras
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sören Auer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim Wittenborg .

Editor information

Editors and Affiliations

  1. University of Salford, Salford, UK

    Apostolos Antonacopoulos

  2. University of Waikato, Hamilton, New Zealand

    Annika Hinze

  3. Sorbonne University (CNRS), Paris, France

    Benjamin Piwowarski

  4. University of La Rochelle (L3i Laboratory), La Rochelle, France

    Mickaël Coustaty

  5. University of Padova, Padua, Italy

    Giorgio Maria Di Nunzio

  6. University of Hamburg, Hamburg, Germany

    Francesco Gelati

  7. University of Waikato, Hamilton, New Zealand

    Nicholas Vanderschantz

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wittenborg, T., Karras, O., Auer, S. (2024). SWARM-SLR - Streamlined Workflow Automation for Machine-Actionable Systematic Literature Reviews. In: Antonacopoulos, A., et al. Linking Theory and Practice of Digital Libraries. TPDL 2024. Lecture Notes in Computer Science, vol 15177. Springer, Cham. https://doi.org/10.1007/978-3-031-72437-4_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72437-4_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72436-7

  • Online ISBN: 978-3-031-72437-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics