Abstract
Machine learning and artificial intelligence methods are increasingly being used in personalized medicine, including precision oncology. Ma et al. (Nature Cancer 2021) have developed a new method called ‘transfer of cell line response prediction’ (TCRP) to train predictors of drug response in cancer cell lines and optimize their performance in higher complex cancer model systems via few-shot learning. TCRP has been presented as a successful modelling approach in multiple case studies. Given the importance of this approach for assisting clinicians in their treatment decision processes, we sought to independently reproduce the authors’ findings and improve the reusability of TCRP in new case studies, including validation in clinical-trial datasets—a high bar for drug-response prediction. Our reproducibility results, while not reaching the same level of superiority as those of the original authors, were able to confirm the superiority of TCRP in the original clinical context. Our reusability results indicate that, in the majority of novel clinical contexts, TCRP remains the superior method for predicting response for both preclinical and clinical settings. Our results thus support the superiority of TCRP over established statistical and machine learning approaches in preclinical and clinical settings. We also developed new resources to increase the reusability of the TCRP model for future improvements and validation studies.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
All datasets used in our study are available on the data platform ORCESTRA, under dataset names GDSC_2020(v1-8.2), GDSC_2020(v2-8.2), gCSI_2019, CTRPv2_2015, PDTX_2019, PDXE, UHNBreast_2019, scRNA_GSE25066_Breast and scRNA_GSE41998_Breast.
Code availability
The code to run and visualize the exact results of our reproducibility attempt as well as all our novel analyses for reusability are available in the corresponding Code Ocean capsule (Version 2.5)22. Code used for the analysis is available at github.com/bhklab/TCRP_Reusability_Report.
References
Trastulla, L., Noorbakhsh, J., Vazquez, F., McFarland, J. & Iorio, F. Computational estimation of quality and clinical relevance of cancer cell lines. Mol. Syst. Biol. 18, e11017 (2022).
Ma, J. et al. Few-shot learning creates predictive models of drug response that translate from high-throughput screens to individual patients. Nat. Cancer 2, 233–244 (2021).
Yang, W. et al. Genomics of Drug Sensitivity in Cancer (GDSC): a resource for therapeutic biomarker discovery in cancer cells. Nucleic Acids Res. 41, D955–D961 (2013).
Bruna, A. et al. A biobank of breast cancer explants with preserved intra-tumor heterogeneity to screen anticancer compounds. Cell 167, 260–274 (2016).
Smirnov, P. et al. Evaluation of statistical approaches for association testing in noisy drug screening data. BMC Bioinf. 23, 188 (2022).
Mammoliti, A. et al. Orchestrating and sharing large multimodal data for transparent and reproducible research. Nat. Commun. 12, 5797 (2021).
Seashore-Ludlow, B. et al. Harnessing coÿnnectivity in a large-scale small-molecule sensitivity dataset. Cancer Discov 5, 1210–1223 (2015).
Haverty, P. M. et al. Reproducible pharmacogenomic profiling of cancer cell line panels. Nature 533, 333–337 (2016).
Iorio, F. et al. A landscape of pharmacogenomic interactions in cancer. Cell 166, 740–754 (2016).
Safikhani, Z. et al. Gene isoforms as expression-based biomarkers predictive of drug response in vitro. Nat. Commun. 8, 1126 (2017).
Thu, K. L. et al. Disruption of the anaphase-promoting complex confers resistance to TTK inhibitors in triple-negative breast cancer. Proc. Natl Acad. Sci. USA 115, E1570–E1577 (2018).
Gao, H. et al. High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nat. Med. 21, 1318–1325 (2015).
Hatzis, C. et al. A genomic predictor of response and survival following taxane-anthracycline chemotherapy for invasive breast cancer. JAMA 305, 1873–1881 (2011).
Itoh, M. et al. Estrogen receptor (ER) mRNA expression and molecular subtype distribution in ER-negative/progesterone receptor-positive breast cancers. Breast Cancer Res. Treat. 143, 403–409 (2014).
Baldasici, O. et al. Circulating small EVs miRNAs as predictors of pathological response to neo-adjuvant therapy in breast cancer patients. Int. J. Mol. Sci. 23, 12625 (2022).
Horak, C. E. et al. Biomarker analysis of neoadjuvant doxorubicin/cyclophosphamide followed by ixabepilone or Paclitaxel in early-stage breast cancer. Clin. Cancer Res. 19, 1587–1595 (2013).
RECIST 1.1 (EORTC); https://recist.eortc.org/recist-1-1-2/
Safikhani, Z. et al. Revisiting inconsistency in large pharmacogenomic studies. F1000Res. 5, 2333 (2016).
Barretina, J. et al. The Cancer Cell Line Encyclopedia enables predictive modelling of anticancer drug sensitivity. Nature 483, 603–607 (2012).
Raff, E. Research reproducibility as a survival analysis. In Proc. AAAI Conference on Artificial Intelligence Vol. 35, 469–478 (AAAI, 2021).
Clyburne-Sherin, A., Fei, X. & Green, S. A. Computational reproducibility via containers in psychology. Meta-Psychology https://doi.org/10.15626/MP.2018.892 (2019).
Reusability Report: Few-shot learning creates predictive models of drug response that translate from high-throughput screens to individual patients (Version 2.5) (Code Ocean); https://codeocean.com/capsule/8411716/tree/v2
Acknowledgements
This work was supported by Canadian Cancer Society Data Transformations (707609) and by a Natural Sciences and Engineering Research Council of Canada Discovery Grant (RGPIN-2021-02680).
Author information
Authors and Affiliations
Contributions
E.S. and F.Y. conducted the reproducibility and reusability experiments, curated datasets, drafted the first version of the manuscript and integrated all the edits. B.W. supervised the manuscript. B.H.-K. contributed ideas and supervised the manuscript writing and final edits.
Corresponding author
Ethics declarations
Competing interests
B.H.K. is a shareholder and paid consultant for Code Ocean Inc. E.S. is a paid consultant for Code Ocean Inc. The remaining authors declare no competing interests.
Peer review
Peer review information
Nature Machine Intelligence thanks Bo Yuan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Tables 1–8.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
So, E., Yu, F., Wang, B. et al. Reusability report: Evaluating reproducibility and reusability of a fine-tuned model to predict drug response in cancer patient samples. Nat Mach Intell 5, 792–798 (2023). https://doi.org/10.1038/s42256-023-00688-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-023-00688-4
