Skip to main content
SpringerLink
Log in

A Study on a Stopping Strategy for Systematic Reviews Based on a Distributed Effort Approach

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12260))

Abstract

Systematic reviews are scientific investigations that use strategies to include a comprehensive search of all potentially relevant articles and the use of explicit, reproducible criteria in the selection of articles for review. As time and resources are limited for compiling a systematic review, limits to the search are needed. In this paper, we describe the stopping strategy that we have been designed and refined over three years of participation to the CLEF eHealth Technology Assisted Review Task. In particular, we present a comparison of a Continuous Active Learning approach that uses either a fixed amount or a variable amount of resources according to the size of the pool. The results show that our approach performs on average much better than any other participant in the CLEF 2019 eHealth TAR task. Nevertheless, a failure analysis allows to understand the weak points of this approach and possible future directions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.clef-initiative.eu.

  2. 2.

    https://www.ncbi.nlm.nih.gov/pubmed/.

  3. 3.

    We used the non-exact version given the ties. https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/wilcox.test.

References

  1. Cook, D.J., Mulrow, C.D., Haynes, R.B.: Systematic reviews: synthesis of best evidence for clinical decisions. Ann. Intern. Med. 126(5), 376–380 (1997)

    Article  Google Scholar 

  2. Cormack, G.V., Grossman, M.R.: Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. In: Geva, S., Trotman, A., Bruza, P., Clarke, C.L., Järvelin, K. (eds.) The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014, Gold Coast , QLD, Australia, 06–11 July 2014, pp. 153–162. ACM (2014)

    Google Scholar 

  3. Goeuriot, L., et al.: CLEF 2017 eHealth evaluation lab overview. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 291–303. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_26

    Chapter  Google Scholar 

  4. Gopalakrishnan, S., Ganeshkumar, P.: Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. J. Fam. Med. Primary Care 2(1), 9–14 (2013)

    Article  Google Scholar 

  5. Grossman, M.R., Cormack, G.V., Roegiest, A.: TREC 2016 total recall track overview. In: Proceedings of The Twenty-Fifth TREC 2016, Gaithersburg, Maryland, USA, 15–18 November 2016 (2016)

    Google Scholar 

  6. Higgins, J.P.T. (eds.): Cochrane Handbook for Systematic Reviews of Interventions, version 6.9 edn., Cochrane, July 2019. www.training.cochrane.org/handbook

  7. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. In: Cappellato, L., Ferro, N., Goeuriot, L., Mandl, T. (eds.) CEUR Workshop Proceedings of Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, vol. 1866, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017)

    Google Scholar 

  8. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. In: CEUR Workshop Proceedings of Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017)

    Google Scholar 

  9. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2018 technologically assisted reviews in empirical medicine overview. In: Cappellato, L., Ferro, N., Nie, J.-Y., Soulier, L., eds. CEUR Workshop Proceedings of Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, 10–14 September 2018, vol. 2125. CEUR-WS.org (2018)

    Google Scholar 

  10. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2019 technology assisted reviews in empirical medicine overview. In: CEUR Workshop Proceedings of CLEF 2019 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS.org (2019)

    Google Scholar 

  11. Kastner, M., Straus, S., Goldsmith, C.H.: Estimating the horizon of articles to decide when to stop searching in systematic reviews: an example using a systematic review of RCTS evaluating osteoporosis clinical decision support tools. In: Proceedings of AMIA Annual Symposium. AMIA Symposium, vol. 2007, no. 10, pp. 389–393 (2007)

    Google Scholar 

  12. Kelly, L., et al.: Overview of the CLEF eHealth evaluation lab 2019. In: Crestani, F., et al. (eds.) CLEF 2019. LNCS, vol. 11696, pp. 322–339. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28577-7_26

    Chapter  Google Scholar 

  13. Li, D., Kanoulas, E.: Automatic thresholding by sampling documents and estimating recall. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) CEUR Workshop Proceedings of Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, 9–12 September 2019, vol. 2380. CEUR-WS.org (2019)

    Google Scholar 

  14. Nunzio, G.M.D.: A new decision to take for cost-sensitive Naïve Bayes classifiers. Inf. Process. Manag. 50(5), 653–674 (2014)

    Article  Google Scholar 

  15. Nunzio, G.M.D.: Interactive text categorisation: the geometry of likelihood spaces. Stud. Comput. Intell. 668, 13–34 (2017)

    Google Scholar 

  16. Di Nunzio, G.M.: A study of an automatic stopping strategy for technologically assisted medical reviews. In: Proceedings of Advances in Information Retrieval - 40th European Conference on IR Research, ECIR 2018, Grenoble, France , 26–29 March 2018, pp. 672–677 (2018)

    Google Scholar 

  17. Di Nunzio, G.M., Ciuffreda, G., Vezzani, F.: Interactive sampling for systematic reviews. IMS Unipd at CLEF 2018 ehealth task 2. In: Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, 10–14 September 2018 (2018)

    Google Scholar 

  18. Di Nunzio, G.M., Maistro, M., Vezzani, F.: A gamified approach to Naïve Bayes classification: a case study for newswires and systematic medical reviews. In: Companion of the Web Conference 2018 on the Web Conference 2018, WWW 2018, Lyon, France, 23–27 April 2018, pp. 1139–1146 (2018)

    Google Scholar 

  19. Di Nunzio, G.M., Maistro, M., Zilio, D.: Gamification for machine learning: the classification game. In: Proceedings of the Third International Workshop on Gamification for Information Retrieval co-located with 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016), pp. 45–52, Pisa, Italy, 21 July 2016 (2016)

    Google Scholar 

  20. Di Nunzio, G.M., Maistro, M., Zilio, D.: The University of Padua (IMS) at TREC 2016 total recall track. In: Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, 15–18 November 2016 (2016)

    Google Scholar 

  21. Robertson, S.E.: The Probability Ranking Principle in IR, pp. 281–286. Morgan Kaufmann Publishers Inc., San Francisco (1997)

    Google Scholar 

  22. Roegiest, A., Cormack, G.V., Grossman, M.R., Clarke, C.L.A.: TREC 2015 total recall track overview. In: Proceedings of The Twenty-fourth TREC 2015, Gaithersburg, Maryland, USA, 17–20 November 2015 (2015)

    Google Scholar 

  23. Scells, H., Zuccon, G., Koopman, B., Clark, J.: Automatic boolean query formulation for systematic review literature search. In: Huang, Y., King, I., Liu, T.-Y., van Steen, M. (eds.) WWW 2020: The Web Conference 2020, Taipei, Taiwan, 20–24 April 2020, pp. 1071–1081. ACM/IW3C2 (2020)

    Google Scholar 

  24. Williams, V., Boylan, A.-M., Nunan, D.: Qualitative research as evidence: expanding the paradigm for evidence-based healthcare. BMJ Evid.-Based Med. 24(5), 168–169 (2019)

    Article  Google Scholar 

  25. Xiang, Y.-T., et al.: Timely research papers about COVID-19 in China. Lancet 395(10225), 684–685 (2020)

    Article  Google Scholar 

  26. Zhang, H., Abualsaud, M., Ghelani, N., Smucker, M.D., Cormack, G.V., Grossman, M.R.: Effective user interaction for high-recall retrieval: less is more. In: Cuzzocrea, A., (eds.) Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, 22–26 October 2018, pp. 187–196. ACM (2018)

    Google Scholar 

  27. Zou, J., Li, D., Kanoulas, E.: Technology assisted reviews: Finding the last few relevant documents by asking yes/no questions to reviewers. In: Thompson, K.C., Mei, Q., Brian, D.D., Liu, Y., Yilmaz, E. (eds.) The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, 08–12 July 2018, pp. 949–952. ACM (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Engineering, University of Padua, Padua, Italy

    Giorgio Maria Di Nunzio

  2. Department of Mathematics, University of Padua, Padua, Italy

    Giorgio Maria Di Nunzio

Authors
  1. Giorgio Maria Di Nunzio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giorgio Maria Di Nunzio .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering, Democritus University of Thrace, Xanthi, Greece

    Avi Arampatzis

  2. University of Amsterdam, Amsterdam, The Netherlands

    Evangelos Kanoulas

  3. Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

    Theodora Tsikrika

  4. Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

    Stefanos Vrochidis

  5. Faculty of Library, Information and Media Science, University of Tsukuba, Ibaraki, Japan

    Hideo Joho

  6. Department of Computer Science, University of Copenhagen, Copenhagen, Denmark

    Christina Lioma

  7. Brown University, Providence, RI, USA

    Carsten Eickhoff

  8. LIMSI-CNRS, Orsay, France

    Aurélie Névéol

  9. Department of Information Engineering, University of Padova, Padua, Italy

    Linda Cappellato

  10. Department of Information Engineering, University of Padova, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Di Nunzio, G.M. (2020). A Study on a Stopping Strategy for Systematic Reviews Based on a Distributed Effort Approach. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2020. Lecture Notes in Computer Science(), vol 12260. Springer, Cham. https://doi.org/10.1007/978-3-030-58219-7_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58219-7_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58218-0

  • Online ISBN: 978-3-030-58219-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation