Maciej Rybinski
Sarvnaz Karimi
ABSTRACT
High-quality evidence from the biomedical literature is crucial for decision making of oncologists who treat cancer patients. Search for evidence on a specific treatment for a patient is the challenge set by the precision medicine track of TREC in 2020. To address this challenge, we propose a two-step method to incorporate treatment into the query formulation and ranking. Training of such ranking function uses a zero-shot setup to incorporate the novel focus on treatments which did not exist in any of the previous TREC tracks. Our treatment-aware neural reranking approach, FAT, achieves state-of-the-art effectiveness for TREC Precision Medicine 2020. Our analysis indicates that the BERT-based rerankers automatically learn to score documents through identifying concepts relevant to precision medicine, similar to hand-crafted heuristics successful in the earlier studies.
Supplemental Material
- 2021. BioBERT @ Huggingface model repository. https://huggingface.co/ monologg/biobert_v1.0_pubmed_pmc. Accessed: 2021-02-22.Google Scholar
- 2021. DrugBank. https://go.drugbank.com/. Accessed: 2021-02-22.Google Scholar
- 2021. Transformers. https://huggingface.co/transformers/. Accessed: 2021-02-22Google Scholar
- Maristella Agosti, Giorgio Maria Di Nunzio, and Stefano Marchesin. 2019. An Analysis of Query Reformulation Techniques for Precision Medicine. In SIGIR. Paris, France, 973--976. Google ScholarDigital Library
- Gianni Amati and Cornelis Joost Van Rijsbergen. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. TOIS, Vol. 20, 4 (2002), 357--389. Google ScholarDigital Library
- Hilda Bastian, Paul Glasziou, and Iain Chalmers. [n.d.]. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? ( n.,d.]).Google Scholar
- Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: Pretrained Language Model for Scientific Text. In EMNLP.Google Scholar
- Mette Eriksen and Tove Frandsen. 2018. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. Journal of the Medical Library Association, Vol. 106, 4 (2018), 420--431.Google ScholarCross Ref
- Erik Faessler, Michel Oleynik, and Udo Hahn. 2020. What Makes a Top-Performing Precision Medicine Search Engine? Tracing Main System Features in a Systematic Way. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 459--468. Google ScholarDigital Library
- Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. 2020. A Deep Look into neural ranking models for information retrieval. Information Processing & Management, Vol. 57, 6 (2020), 102067.Google ScholarCross Ref
- Gordon Guyatt, John Cairns, David Churchill, Deborah Cook, Brian Haynes, Jack Hirsh, Jan Irvine, Mark Levine, Mitchell Levine, Jim Nishikawa, David Sackett, Patrick Brill-Edwards, Hertzel Gerstein, Jim Gibson, Roman Jaeschke, Anthony Kerigan, Alan Neville, Akbar Panju, Allan Detsky, Murray Enkin, Pamela Frid, Martha Gerrity, Andreas Laupacis, Valerie Lawrence, Joel Menard, Virginia Moyer, Cynthia Mulrow, Paul Links, Andrew Oxman, Jack Sinclair, and Peter Tugwell. 1992. Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine. JAMA, Vol. 268, 17 (1992), 2420--2425.Google ScholarCross Ref
- William Hersh, Ravi Teja Bhupatiraju, and Sarah Corley. 2004. Enhancing access to the Bibliome: the TREC Genomics Track. Studies in Health Technology and Informatics, Vol. 107, Pt 2 (2004), 773--777.Google Scholar
- William Hersh and Ellen Voorhees. 2009. TREC genomics special issue overview. Information Retrieval, Vol. 12 (2009), 1--15. Google ScholarDigital Library
- Jon J Hiles and Jill M Kolesar. 2008. Role of sunitinib and sorafenib in the treatment of metastatic renal cell carcinoma. American Journal of Health-System Pharmacy, Vol. 65, 2 (2008), 123--131.Google ScholarCross Ref
- Xiaoli Huang, Jimmy Lin, and Dina Demner-Fushman. 2006. Evaluation of PICO as a knowledge representation for clinical questions. In AMIA Annual Symposium proceedings. 359--363.Google Scholar
- Su Nam Kim, David Martinez, Lawrence Cavedon, and Lars Yenken. 2011. Automatic classification of sentences to support Evidence Based Medicine. BMC Bioinformatics, Vol. 12, S5 (2011).Google ScholarCross Ref
- Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 951--958.Google ScholarCross Ref
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, Vol. 36, 4 (09 2019), 1234--1240.Google Scholar
- Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained transformers for text ranking: BERT and beyond. arXiv preprint arXiv:2010.06467 (2020).Google Scholar
- Xiaofeng Liu, Lu Li, Zuoxi Yang, and Shoubin Dong. 2019. SCUT-CCNL at TREC 2019 Precision Medicine Track. In TREC. Gaithersburg, MD.Google Scholar
- Sean MacAvaney, Arman Cohan, and Nazli Goharian. 2020. SLEDGE: A Simple Yet Effective Baseline for COVID-19 Scientific Knowledge Search. arxiv: 2005.02365 [cs.IR]Google Scholar
- David Martinez, Sarvnaz Karimi, Lawrence Cavedon, and Timothy Baldwin. 2008. Facilitating biomedical systematic reviews using ranked text retrieval and classification. In Australasian Document Computing Symposium. 53--60.Google Scholar
- Ryan McDonald, George Brokos, and Ion Androutsopoulos. 2018. Deep Relevance Ranking Using Enhanced Document-Query Interactions. In EMNLP. Brussels, Belgium, 1849--1860.Google ScholarCross Ref
- Lowell K Milliken, Sirisha K Motomarry, and Anagha Kulkarni. 2019. ARtPM: article retrieval for precision medicine. Journal of biomedical informatics, Vol. 95 (2019), 103224.Google ScholarDigital Library
- Vincent Nguyen, Maciek Rybinski, Sarvnaz Karimi, and Zhenchang Xing. 2020. Pandemic Literature Search: Finding Information on COVID-19. In Proceedings of the The 18th Annual Workshop of the Australasian Language Technology Association. 92--97.Google Scholar
- NLM. 2021. Medline - NLM. https://www.nlm.nih.gov/medline/. [Online; accessed 26-Feb-2021].Google Scholar
- Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv:1901.04085 (2019). arxiv: 1901.04085 [cs.IR]Google Scholar
- Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. arXiv preprint arXiv:1910.14424 (2019).Google Scholar
- U.S National Library of Medicine. 2017. https://meshb.nlm.nih.gov/#/fieldSearch.Google Scholar
- Cedric Panje, Markus Glatzer, Charlotta Siren, Ludwig Plasswilm, and Paul Putora. 2018. Treatment Options in Oncology. JCO Clinical Cancer Informatics (2018).Google Scholar
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135--1144. Google ScholarDigital Library
- W. Scott Richardson, Mark Wilson, Jim Nishikawa, and Robert Hayward. 1995. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club, Vol. 123, 3 (1995), A12--3.Google ScholarCross Ref
- Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, and William Hersh. 2020. TREC-COVID: Rationale and Structure of an Information Retrieval Shared Task for COVID-19. The Journal of the American Medical Informatics Association, Vol. 27, 9 (2020), 1431--1436.Google ScholarCross Ref
- Kirk Roberts, Dina Demner-Fushman, Ellen Voorhees, William R. Hersh, Steven Bedrick, Alexander Lazar, and Shubham Pant. 2017. Overview of the TREC 2017 Precision Medicine Track. In TREC. Gaithersburg, MD.Google Scholar
- Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, Steven Bedrick, and William R. Hersh. 2021. Overview of the TREC 2020 Precision Medicine Track. In (To appear in) TREC. Gaithersburg, MD.Google Scholar
- Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, William R. Hersh, Steven Bedrick, and Alexander J. Lazar. 2018. Overview of the TREC 2018 Precision Medicine Track. In TREC. Gaithersburg, MD.Google Scholar
- Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, William R. Hersh, Steven Bedrick, Alexander J. Lazar, Shubham Pant, and Funda Meric-Bernstam. 2019. Overview of the TREC 2019 Precision Medicine Track. In TREC. Gaithersburg, MD.Google Scholar
- K. Roberts, M. Simpson, D. Demner-Fushman, E. Voorhees, and W. Hersh. 2016. State-of-the-art in Biomedical Literature Retrieval for Clinical Cases: A Survey of the TREC 2014 CDS Track. Information Retrieval, Vol. 19, 1--2 (2016), 113--148. Google ScholarDigital Library
- Kirk Roberts, Matthew S. Simpson, Ellen Voorhees, and William R. Hersh. 2015. Overview of the TREC 2015 Clinical Decision Support Track. In Text REtrieval Conference. Gaithersburg, MD.Google Scholar
- Stephen Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1995. Okapi at TREC-3. In TREC. Gaithersburg, MD, US. https://trec.nist.gov/pubs/trec3/t3_proceedings.htmlGoogle Scholar
- Maciej Rybinski and Sarvnaz Karimi. 2020. CSIROmed at 2020 TREC Precision Medicine Track. In TREC. Online.Google Scholar
- Maciej Rybinski, Sarvnaz Karimi, and Cecile Paris. 2019. CSIRO at 2019 TREC Precision Medicine Track. In TREC. Gaithersburg, MD.Google Scholar
- Maciej Rybinski, Jerry Xu, and Sarvnaz Karimi. 2020. Clinical trial search: Using biomedical language understanding models for re-ranking. Journal of Biomedical Informatics, Vol. 109 (2020), 103530.Google ScholarDigital Library
- Ellen Voorhees, Alam Tasmeer, Demner-Fushman Dina, Hersh William, and Kyle Lo. 2020. TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection. ACM SIGIR Forum, Vol. 54, 1 (2020), 1--12. Google ScholarDigital Library
- Emine Yilmaz, Evangelos Kanoulas, and Javed A Aslam. 2008. A Simple and Efficient Sampling Method for Estimating AP and NDCG. In SIGIR. Singapore, 603--610. Google ScholarDigital Library
- Xuesi Zhou, Xin Chen, Jian Song, Gang Zhao, and Ji Wu. 2018. Team Cat-Garfield at TREC 2018 Precision Medicine Track. In TREC,, Ellen M. Voorhees and Angela Ellis (Eds.). Gaithersburg, MD.Google Scholar
- Huijia Zhu, Ni Yuan, Cai Peng, Qiu Zhaoming, and Cao Feng. 2012. Automatic extracting of patient-related attributes: disease, age, gender and race. Studies in health technology and informatics, Vol. 180 (2012), 589--593.Google Scholar
Index Terms
- Will Sorafenib Help?: Treatment-aware Reranking in Precision Medicine Search
Recommendations
Science2Cure: A Clinical Trial Search Prototype
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information RetrievalWith the advances in precision medicine, identifying clinical trials relevant to a specific patient profile becomes more challenging. Often very specific molecular-level patient features need to be matched for the trial to be deemed relevant. Clinical ...
A2A-API: A Prototype for Biomedical Information Retrieval Research and Benchmarking
SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information RetrievalFinding relevant literature is crucial for biomedical research and in the practice of evidence-based medicine, making biomedical search an important application area within the field of information retrieval. This is recognised by the broader IR ...
A Self-Learning Resource-Efficient Re-Ranking Method for Clinical Trials Search
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge ManagementComplex search scenarios, such as those in biomedical settings, can be challenging. One such scenario is matching a patient's profile to relevant clinical trials. There are multiple criteria that should match for a document (clinical trial) to be ...
Comments