Abstract
Purpose
Rapid evaporative ionization mass spectrometry (REIMS) is an emerging technology for clinical margin detection. Deployment of REIMS depends on construction of reliable deep learning models that can categorize tissue according to its metabolomic signature. Challenges associated with developing these models include the presence of noise during data acquisition and the variance in tissue signatures between patients. In this study, we propose integration of uncertainty estimation in deep models to factor predictive confidence into margin detection in cancer surgery.
Methods
iKnife is used to collect 693 spectra of cancer and healthy samples acquired from 91 patients during basal cell carcinoma resection. A Bayesian neural network and two baseline models are trained on these data to perform classification as well as uncertainty estimation. The samples with high estimated uncertainty are then removed, and new models are trained using the clean data. The performance of proposed and baseline models, with different ratios of filtered data, is then compared.
Results
The data filtering does not improve the performance of the baseline models as they cannot provide reliable estimations of uncertainty. In comparison, the proposed model demonstrates a statistically significant improvement in average balanced accuracy (75.2%), sensitivity (74.1%) and AUC (82.1%) after removing uncertain training samples. We also demonstrate that if highly uncertain samples are predicted and removed from the test data, sensitivity further improves to 88.2%.
Conclusions
This is the first study that applies uncertainty estimation to inform model training and deployment for tissue recognition in cancer surgery. Uncertainty estimation is leveraged in two ways: by factoring a measure of input noise in training the models and by including predictive confidence in reporting the outputs. We empirically show that considering uncertainty for model development can help improve the overall accuracy of a margin detection system using REIMS.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Manoli S-M, Moutsoudis A, Papageorgiou C, Lallas K, Rigas H-M, Kyrmanidou E, Papadimitriou I, Paschou E, Spyridis I, Gkentsidi T, Sotiriou E, Vakirlis E, Ioannidis D, Apalla Z, Lallas A (2020) Real-life data on basal cell carcinoma treatment: insights on clinicians’ therapeutic choices from an institutional hospital registry. Dermatol Ther 33(6):14414
Filho RB, de Carvalho Fantini B, Dos Santos CA, Melo RV, Rosan I, Chahud F, da Silva Souza C (2019) Attributes and risk factors of positive margins on 864 excisions of basal cell carcinomas: a single-center retrospective study. J Dermatol Treat 31(6):589–596
Balog J, Sasi-Szabo L, Kinross J, Lewis MR, Muirhead LJ, Veselkov K, Mirnezami R, Dezso B (2013) Intraoperative tissue identification using rapid evaporative ionization mass spectrometry. Sci Transl Med 5(2):194
Santilli A, Jamzad A, Janssen N, Kaufmann M, Connolly L, Vanderbeck K, Wang A, McKay D, Rudan J, Fichtinger G, Mousavi P (2020) Perioperative margin detection in bcc using a deep learning framework: a feasibility study. Int J CARS 15:887–96
Jamzad A, Sedghi A, Santilli AML, Janssen NNY, Kaufmann M, Ren KYM, Vanderbeck K, Wang A, Mckay D, Rudan JF, Fichtinger G, Mousavi P (2020) Improved resection margins in surgical oncology using intraoperative mass spectrometry. In: Medical image computing and computer assisted intervention, MICCAI, Lecture notes in computer science, vol 12263. Springer, Cham https://doi.org/10.1007/978-3-030-59716-0_5
Wells G, Prest H, William C, Iv R. Application note chemical analysis signal, noise, and detection limits in mass spectrometry
Loquercio A, Segu M, Scaramuzza D (2020) A general framework for uncertainty estimation in deep learning. IEEE Robotics Autom Lett 5(2):3153–3160
Vranken JF, van de Leur RR, Gupta DK, Juarez Orozco LE, Hassink RJ, van der Harst P, Doevendans PA, Gulshad S, van Es R (2021) Uncertainty estimation for deep learning-based automated analysis of 12-lead electrocardiograms. Eur Heart J Digital Health 2(3):401–415
Gawlikowski J, Tassi CRN, Ali M, Lee J, Humt M, Feng J, Kruspe A, Triebel R, Jung P, Roscher R et al (2021) A survey of uncertainty in deep neural networks. arXiv:2107.03342
Kendall A, Gal Y (2017) What uncertainties do we need in Bayesian deep learning for computer vision? arXiv:1703.04977
Hüllermeier E, Waegeman W (2021) Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach Learn 110(3):457–506
DeVries T, Taylor GW (2018) Leveraging uncertainty estimates for predicting segmentation quality. arXiv:1807.00502
Senapati J, Roy AG, Pölsterl S, Gutmann D, Gatidis S, Schlett C, Peters A, Bamberg F, Wachinger C (2020) Bayesian neural networks for uncertainty estimation of imaging biomarkers. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 12436. LNCS, pp 270–280
Ruhe D, Cinà G, Tonutti M, de Bruin D, Elbers P (2019) Bayesian modelling in practice: using uncertainty to improve trustworthiness in medical applications. arXiv:1906.08619
Gal Y, Ghahramani Z (2016) Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: International conference on machine learning, pp 1050–1059
Murphy KP (2022) Probabilistic machine learning: advanced topics. MIT Press, Cambridge
St-John ER, Al-Khudairi R, Ashrafian H, Athanasiou T, Takats Z, Hadjiminas DJ, Darzi A, Leff DR (2017) Diagnostic accuracy of intraoperative techniques for margin assessment in breast cancer surgery. Anal Surg 265(2):300–310
Funding
We would like to thank the following sources of funding: Natural Sciences and Engineering Council of Canada (NSERC), the Canadian Institute for Health Research (CIHR), Southeastern Ontario Academic Medical Organization (SEAMO) Innovation Fund, Britton Smith Chair in Surgery to J. Rudan, and Canada Research Chair to G. Fichtinger.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare no conflicts of interest.
Ethical approval
This study was approved by the Queen’s University Health Sciences Research Ethics Board.
Informed consent
All patients that participated in the study gave informed verbal and written consent.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Fooladgar, F., Jamzad, A., Connolly, L. et al. Uncertainty estimation for margin detection in cancer surgery using mass spectrometry. Int J CARS 17, 2305–2313 (2022). https://doi.org/10.1007/s11548-022-02764-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-022-02764-3