Abstract
Curated Document Databases play a critical role in helping researchers find relevant articles in available literature. One such database is the ORRCA (Online Resource for Recruitment research in Clinical trials) database. The ORRCA database brings together published work in the field of clinical trials recruitment research into a single searchable collection. Document databases, such as ORRCA, require year-on-year updating as further relevant documents become available on a continuous basis. The updating of curated databases is a labour intensive and time consuming task. Machine learning techniques can help to automate the update process and reduce the workload needed for screening articles for inclusion. This paper presents an automated approach to the updating of ORRCA documents repository. The proposed automated approach is a learning to rank model. The approach is evaluated using the documents in the ORRCA database. Data from the ORRCA original systematic review was used to train the learning to rank model, and data from the ORRCA 2015 and 2017 updates was used to evaluate performance of the model. The evaluation demonstrated that significant resource savings can be made using the proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
Sklearn is a python based machine learning library that contains implementations for various machine learning algorithms.
- 4.
Logarithmic loss (related to cross-entropy) measures the performance of a classification model where the prediction output is a probability value between 0 and 1.
References
Beller, E., et al.: Making progress with the automation of systematic reviews: principles of the international collaboration for the automation of systematic reviews (ICASR). Syst. Rev. 7(1), 1–7 (2018)
Dehghani, M., Zamani, H., Severyn, A., Kamps, J., Croft, W.B.: Neural ranking models with weak supervision. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 65–74 (2017)
Dowell, K.G., McAndrews-Hill, M.S., Hill, D.P., Drabkin, H.J., Blake, J.A.: Integrating text mining into the MGI biocuration workflow. Database J. Biol. Databases Curation (2009)
Howard, B.E., et al.: Swift-review: a text-mining workbench for systematic review. Syst. Rev. 5(1), 87 (2016)
Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. CEUR Workshop Proc. 1866, 1–29 (2017)
Kearney, A., et al.: Development of an online resource for recruitment research in clinical trials to organise and map current literature. Clin. Trials 15(6), 533–542 (2018)
Khabsa, M., Elmagarmid, A., Ilyas, I., Hammady, H., Ouzzani, M.: Learning to identify relevant studies for systematic reviews using random forest and external information. Mach. Learn. 102(3), 465–482 (2015). https://doi.org/10.1007/s10994-015-5535-7
Krallinger, M., et al.: The protein-protein interaction tasks of biocreative iii: classification/ranking of articles and linking bio-ontology concepts to full text. BMC Bioinform. 12(Suppl. 8), S3 (2011)
Lerner, I., Créquit, P., Ravaud, P., Atal, I.: Automatic screening using word embeddings achieved high sensitivity and workload reduction for updating living network meta-analyses. J. Clin. Epidemiol. 108, 86–94 (2019)
MacAvaney, S., Nardini, F.M., Perego, R., Tonellotto, N., Goharian, N., Frieder, O.: Efficient document re-ranking for transformers by precomputing term representations. arXiv preprint arXiv:2004.14255 (2020)
Nogueira, R., Jiang, Z., Lin, J.: Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020)
Norman, C.R., Gargon, E., Leeflang, M.M.G., Névéol, A., Williamson, P.R.: Evaluation of an automatic article selection method for timelier updates of the comet core outcome set database. Database J. Biol. Databases Curation (2019)
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., Ananiadou, S.: Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst. Rev. 4(1), 5 (2015)
Pandey, G.: Utilization of efficient features, vectors and machine learning for ranking techniques. JYU Dissertations (2019)
Pereira, M., Etemad, E., Paulovich, F.: Iterative learning to rank from explicit relevance feedback. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, pp. 698–705 (2020)
Qiao, Y., Xiong, C., Liu, Z., Liu, Z.: Understanding the behaviors of BERT in ranking. arXiv preprint arXiv:1904.07531 (2019)
Suominen, H., et al.: Overview of the CLEF ehealth evaluation lab 2018. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_26
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Muhammad, I., Bollegala, D., Coenen, F., Gamble, C., Kearney, A., Williamson, P. (2020). Maintaining Curated Document Databases Using a Learning to Rank Model: The ORRCA Experience. In: Bramer, M., Ellis, R. (eds) Artificial Intelligence XXXVII. SGAI 2020. Lecture Notes in Computer Science(), vol 12498. Springer, Cham. https://doi.org/10.1007/978-3-030-63799-6_26
Download citation
DOI: https://doi.org/10.1007/978-3-030-63799-6_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63798-9
Online ISBN: 978-3-030-63799-6
eBook Packages: Computer ScienceComputer Science (R0)