skip to main content
10.1145/3366423.3380075acmconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article

Sampling Query Variations for Learning to Rank to Improve Automatic Boolean Query Generation in Systematic Reviews

Published: 20 April 2020 Publication History

Abstract

Searching medical literature for synthesis in a systematic review is a complex and labour intensive task. In this context, expert searchers construct lengthy Boolean queries. The universe of possible query variations can be massive: a single query can be composed of hundreds of field-restricted search terms/phrases or ontological concepts, each grouped by a logical operator nested to depths of sometimes five or more levels deep. With the many choices about how to construct a query, it is difficult to both formulate and recognise effective queries. To address this challenge, automatic methods have recently been explored for generating and selecting effective Boolean query variations for systematic reviews. The limiting factor of these methods is that it is computationally infeasible to process all query variations for training the methods. To overcome this, we propose novel query variation sampling methods for training Learning to Rank models to rank queries. Our results show that query sampling methods do directly impact the ability of a Learning to Rank model to effectively identify good query variations. Thus, selecting appropriate query sampling methods is a key problem for the automatic reformulation of effective Boolean queries for systematic review literature search. We find that the best sampling strategies are those which balance the diversity of queries with the quantity of queries.

References

[1]
Mustafa Abualsaud, Nimesh Ghelani, Haotian Zhang, Mark D Smucker, Gordon V Cormack, and Maura R Grossman. 2018. A system for efficient high-recall retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1317–1320.
[2]
Amal Alharbi, William Briggs, and Mark Stevenson. 2018. Retrieving and ranking studies for systematic reviews: University of Sheffield’s approach to CLEF eHealth 2018 Task 2. In CEUR Workshop Proceedings, Vol. 2125. CEUR Workshop Proceedings.
[3]
Amal Alharbi and Mark Stevenson. 2017. Ranking Abstracts to Identify Relevant Evidence for Systematic Reviews: The University of Sheffield’s Approach to CLEF eHealth 2017 Task 2. In CLEF (Working Notes).
[4]
David Arthur and Sergei Vassilvitskii. 2007. k-means++: The advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms. Society for Industrial and Applied Mathematics, 1027–1035.
[5]
Javed A Aslam, Evangelos Kanoulas, Virgil Pavlu, Stefan Savev, and Emine Yilmaz. 2009. Document selection methodologies for efficient and effective learning-to-rank. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. ACM, 468–475.
[6]
Gabriele Capannini, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, and Nicola Tonellotto. 2016. Quality versus efficiency in document scoring with learning-to-rank models. Information Processing & Management 52, 6 (2016), 1161–1177.
[7]
Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 335–336.
[8]
Claudio Carpineto and Giovanni Romano. 2012. A survey of automatic query expansion in information retrieval. ACM Computing Surveys (CSUR) 44, 1 (2012), 1.
[9]
Jiayi Chen, Su Chen, Yang Song, Hongyu Liu, Yueyao Wang, Qinmin Hu, Liang He, and Yan Yang. 2017. ECNU at 2017 eHealth Task 2: Technologically Assisted Reviews in Empirical Medicine. In CLEF (Working Notes).
[10]
Gordon V Cormack and Maura R Grossman. 2015. Autonomy and reliability of continuous active learning for technology-assisted review. arXiv preprint arXiv:1504.06868(2015).
[11]
Pinar Donmez and Jaime G Carbonell. 2008. Optimizing estimated loss reduction for active sampling in rank learning. In Proceedings of the 25th international conference on Machine learning. ACM, 248–255.
[12]
Kalervo Järvelin and Jaana Kekäläinen. 2000. IR evaluation methods for retrieving highly relevant documents. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 41–48.
[13]
Evangelos Kanoulas, Stefan Savev, Pavel Metrikov, Virgil Pavlu, and Javed Aslam. 2011. A large-scale study of the effect of training set characteristics over learning-to-rank algorithms. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. ACM, 1243–1244.
[14]
S. Karimi, S. Pohl, F. Scholer, L. Cavedon, and J. Zobel. 2010. Boolean versus ranked querying for biomedical systematic reviews. BMC MIDM 10, 1 (2010), 1.
[15]
Athanasios Lagopoulos, Antonios Anagnostou, Adamantios Minas, and Grigorios Tsoumakas. 2018. Learning-to-Rank and Relevance Feedback for Literature Appraisal in Empirical Medicine. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 52–63.
[16]
Grace E. Lee and Aixin Sun. 2018. Seed-driven Document Ranking for Systematic Reviews in Evidence-Based Medicine. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval(SIGIR ’18). ACM, New York, NY, USA, 455–464. https://doi.org/10.1145/3209978.3209994
[17]
Daniel Locke, Guido Zuccon, and Harrisen Scells. 2017. Automatic Query Generation from Legal Texts for Case Law Retrieval. In Asia Information Retrieval Symposium. Springer, 181–193.
[18]
Claudio Lucchese, Franco Maria Nardini, Raffaele Perego, and Salvatore Trani. 2017. The Impact of Negative Samples on Learning to Rank. In LEARNER.
[19]
Jessie McGowan and Margaret Sampson. 2005. Systematic reviews need systematic searchers (IRP). Journal of the Medical Library Association 93, 1 (2005), 74.
[20]
Faith McLellan. 2001. 1966 and all that—when is a literature search done?The Lancet 358, 9282 (2001), 646.
[21]
Rishabh Mehrotra and Emine Yilmaz. 2015. Representative & informative query selection for learning to rank using submodular functions. In Proceedings of the 38th international ACM sigir conference on research and development in information retrieval. ACM, 545–554.
[22]
Mandar Mitra, Amit Singhal, and Chris Buckley. 1998. Improving automatic query expansion. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 206–214.
[23]
M. Miwa, J. Thomas, A. O’Mara-Eves, and S. Ananiadou. 2014. Reducing systematic review workload through certainty-based screening. JBI 51(2014), 242–253.
[24]
Douglas W Oard, Jason R Baron, Bruce Hedin, David D Lewis, and Stephen Tomlinson. 2010. Evaluation of information retrieval for E-discovery. Artificial Intelligence and Law 18, 4 (2010), 347–386.
[25]
Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. 2015. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic reviews 4, 1 (2015), 5.
[26]
John Rathbone. 2017. Automating systematic reviews.Ph.D. Dissertation. Bond University.
[27]
E Sayers and V Miller. 2010. Entrez programming utilities help [internet]. National Center for Biotechnology Information (US).
[28]
Harrisen Scells and Guido Zuccon. 2018. Generating Better Queries for Systematic Reviews. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval(SIGIR ’18). ACM, New York, NY, USA, 475–484.
[29]
Harrisen Scells, Guido Zuccon, Anthony Deacon, and Bevan Koopman. 2017. QUT ielab at CLEF eHealth 2017 technology assisted reviews track: Initial experiments with learning to rank. In CEUR Workshop Proceedings: Working Notes of CLEF 2017: Conference and Labs of the Evaluation Forum, Vol. 1866. CEUR Workshop Proceedings, Paper–98.
[30]
Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2019. Automatic Boolean Query Refinement for Systematic Review Literature Search. In Proceedings of the 2019 World Wide Web Conference.
[31]
Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Shlomo Geva, and Leif Azzopardi. 2017. A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic Reviews. In SIGIR’2017.
[32]
Ian Shemilt, Nada Khan, Sophie Park, and James Thomas. 2016. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic reviews 5, 1 (2016), 140.
[33]
James Thomas and Angela Harden. 2008. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC medical research methodology 8, 1 (2008), 45.
[34]
Mercedes Torres Torres and Clive E Adams. 2017. RevManHAL: towards automatic text generation in systematic reviews. Systematic Reviews 6, 1 (2017), 27.
[35]
G. Tsafnat, P. Glasziou, M.K. Choong, A. Dunn, F. Galgani, and E. Coiera. 2014. Systematic review automation technologies. SR 3, 1(2014), 74.
[36]
Suzan Verberne, Jiyin He, Udo Kruschwitz, Birger Larsen, Tony Russell-Rose, and Arjen P De Vries. 2018. First International Workshop on Professional Search (ProfS2018). In SIGIR. 1431–1434.
[37]
Byron C Wallace, Joël Kuiper, Aakash Sharma, Mingxi Brian Zhu, and Iain J Marshall. 2016. Extracting PICO sentences from clinical trial reports using supervised distant supervision. Journal of Machine Learning Research 17, 132 (2016), 1–25.
[38]
Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. 2017. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision. 2840–2848.
[39]
Huaying Wu, Tingting Wang, Jiayi Chen, Su Chen, Qinmin Hu, and Liang He. 2018. Ecnu at 2018 ehealth task 2: Technologically assisted reviews in empirical medicine. Methods 4, 5 (2018), 7.
[40]
Qiang Wu, Christopher JC Burges, Krysta M Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval 13, 3 (2010), 254–270.
[41]
Jinxi Xu and W Bruce Croft. 1996. Query expansion using local and global document analysis. In Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 4–11.
[42]
Jie Zou, Dan Li, and Evangelos Kanoulas. 2018. Technology Assisted Reviews: Finding the Last Few Relevant Documents by Asking Yes/No Questions to Reviewers. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 949–952.

Cited By

View all
  • (2023)Generating Natural Language Queries for More Effective Systematic Review Screening PrioritisationProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625322(73-83)Online publication date: 26-Nov-2023
  • (2023)Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591703(1426-1436)Online publication date: 19-Jul-2023
  • (2022)From Little Things Big Things GrowProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531748(3176-3186)Online publication date: 6-Jul-2022
  • Show More Cited By

Index Terms

  1. Sampling Query Variations for Learning to Rank to Improve Automatic Boolean Query Generation in Systematic Reviews
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          WWW '20: Proceedings of The Web Conference 2020
          April 2020
          3143 pages
          ISBN:9781450370233
          DOI:10.1145/3366423
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 20 April 2020

          Permissions

          Request permissions for this article.

          Check for updates

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Conference

          WWW '20
          Sponsor:
          WWW '20: The Web Conference 2020
          April 20 - 24, 2020
          Taipei, Taiwan

          Acceptance Rates

          Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)25
          • Downloads (Last 6 weeks)3
          Reflects downloads up to 13 Feb 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2023)Generating Natural Language Queries for More Effective Systematic Review Screening PrioritisationProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625322(73-83)Online publication date: 26-Nov-2023
          • (2023)Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591703(1426-1436)Online publication date: 19-Jul-2023
          • (2022)From Little Things Big Things GrowProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531748(3176-3186)Online publication date: 6-Jul-2022
          • (2022)Search strategy formulation for systematic reviews: Issues, challenges and opportunitiesIntelligent Systems with Applications10.1016/j.iswa.2022.20009115(200091)Online publication date: Sep-2022

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media