Abstract
Automated medical diagnosis systems need to be able to recognize when new diseases emerge, that are not represented in the training data (ID). Even though current out-of-distribution (OOD) detection algorithms can successfully distinguish completely different data sets, they fail to reliably identify samples from novel classes that are similar to the training data. We develop a new ensemble-based procedure that promotes model diversity and exploits regularization to limit disagreement to only OOD samples, using a batch containing an unknown mixture of ID and OOD data. We show that our procedure significantly outperforms state-of-the-art methods, including those that have access, during training, to known OOD data. We run extensive comparisons of our approach on a variety of novel-class detection scenarios, on standard image data sets as well as on new disease detection on medical image data sets (Our code is publicly available at https://github.com/ericpts/reto).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Barbu, A., et al.: ObjectNet: a large-scale bias-controlled dataset for pushing the limits of object recognition models. In: Advances in Neural Information Processing, vol. 32, pp. 9453–9463 (2019)
Beede, E., et al.: A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2020)
Ben-David, S., Blitzer, J., Crammer, K., Pereira, F.: Analysis of representations for domain adaptation. In: Advances in Neural Information Processing Systems, vol. 19, pp. 137–144 (2007)
Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural networks. In: Proceedings of the 32th International Conference on Machine Learning (2015)
Cao, T., Huang, C.W., Hui, D.Y.T., Cohen, J.P.: A benchmark of medical out of distribution detection. arXiv preprint arXiv:2007.04250 (2020)
Chen, Y., Wei, C., Kumar, A., Ma, T.: Self-training avoids using spurious features under domain shift. arXiv preprint arXiv:2006.10032 (2020)
Choi, H., Jang, E., Alemi, A.A.: WAIC, but why? Generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392 (2018)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009)
Fu, Y., Hospedales, T.M., Xiang, T., Gong, S.: Transductive multi-view zero-shot learning. IEEE Trans. Pattern Anal. Mach. Intell. 37, 2332–2345 (2015)
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: Proceedings of Machine Learning Research, vol. 48, pp. 1050–1059 (2016)
Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17, 1–35 (2016)
Geifman, Y., El-Yaniv, R.: Selective classification for deep neural networks. In: Advances in Neural Information Processing Systems, vol. 30, pp. 4878–4887 (2017)
Graves, A.: Practical variational inference for neural networks. In: Advances in Neural Information Processing Systems, vol. 24, pp. 2348–2356 (2011)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Hein, M., Andriushchenko, M., Bitterwolf, J.: Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. In: Proceedings of the International Conference on Learning Representations (2019)
Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. In: Proceedings of the International Conference on Learning Representations (2019)
Katsamenis, I., Protopapadakis, E., Voulodimos, A., Doulamis, A., Doulamis, N.: Transfer learning for COVID-19 pneumonia detection and classification in chest X-ray images. medRxiv (2020)
Kirichenko, P., Izmailov, P., Wilson, A.G.: Why normalizing flows fail to detect out-of-distribution data. In: Advances in Neural Information Processing Systems, vol. 33, pp. 20578–20589 (2020)
Kiryo, R., Niu, G., du Plessis, M.C., Sugiyama, M.: Positive-Unlabeled learning with non-negative risk estimator. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report (2009)
Kumar, A., Ma, T., Liang, P.: Understanding self-training for gradual domain adaptation. arXiv preprint arXiv:2002.11361 (2020)
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in Neural Information Processing Systems, vol. 30, pp. 6402–6413. Curran Associates, Inc. (2017)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. In: Proceedings of the IEEE, pp. 2278–2324 (1998)
Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: Advances in Neural Information Processing Systems, vol. 31, pp. 7167–7177 (2018)
Li, M., Soltanolkotabi, M., Oymak, S.: Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks, pp. 4313–4324. Proceedings of Machine Learning Research (2020)
Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. In: Proceedings of the International Conference on Learning Representations (2018)
Liu, S., Garrepalli, R., Dietterich, T., Fern, A., Hendrycks, D.: Open category detection with PAC guarantees, pp. 3169–3178 (2018)
Lu, A.X., Lu, A.X., Schormann, W., Andrews, D.W., Moses, A.M.: The cells out of sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers. arXiv preprint arXiv:1906.07282 (2019)
Malinin, A., Gales, M.: Predictive uncertainty estimation via prior networks. In: Advances in Neural Information Processing Systems, vol. 32, pp. 7047–7058 (2018)
Nalisnick, E., Matsukawa, A., Teh, Y.W., Gorur, D., Lakshminarayanan, B.: Do deep generative models know what they don’t know? In: Proceedings of the International Conference on Learning Representations (2019)
Neal, R.M.: Bayesian Learning for Neural Networks. Springer, Heidelberg (1996). https://doi.org/10.1007/978-1-4612-0745-0
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)
Ovadia, Y., et al.: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In: Advances in Neural Information Processing Systems, vol. 32, pp. 13991–14002 (2019)
du Plessis, M.C., Niu, G., Sugiyama, M.: Analysis of learning from positive and unlabeled data. In: Advances in Neural Information Processing Systems, vol. 27 (2014)
Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv preprint arXiv:1806.00451 (2018)
Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do ImageNet classifiers generalize to ImageNet? arXiv preprint arXiv:1902.10811 (2019)
Ren, J., Liu, P.J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., Lakshminarayanan, B.: Likelihood ratios for out-of-distribution detection. In: Advances in Neural Information Processing Systems, vol. 32, pp. 14707–14718 (2019)
Sastry, C.S., Oore, S.: Detecting out-of-distribution examples with in-distribution examples and Gram matrices. arXiv preprint arXiv:1912.12510 (2019)
Scott, C., Blanchard, G.: Transductive anomaly detection. Technical report (2008)
Shimodaira, H.: Improving predictive inference under covariate shift by weighting the log-likelihood function. J. Stat. Plan. Infer. 90, 227–244 (2000)
Torralba, A., Fergus, R., Freeman, W.T.: 80 million tiny images: a large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 30, 1958–1970 (2008)
Vapnik, V.N.: Statistical Learning Theory. Wiley, Hoboken (1998)
Wan, Z., Chen, D., Li, Y., Yan, X., Zhang, J., Yu, Y., Liao, J.: Transductive zero-shot learning with visual structure constraint. In: Advances in Neural Information Processing Systems, vol. 32, pp. 9972–9982 (2019)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms (2017)
Yilmaz, F.F., Heckel, R.: Image recognition from raw labels collected without annotators. arXiv preprint arXiv:1910.09055 (2019)
Yu, Q., Aizawa, K.: Unsupervised out-of-distribution detection by maximum classifier discrepancy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Proceedings of the British Machine Vision Conference (BMVC) (2016)
Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Țifrea, A., Stavarache, E., Yang, F. (2021). Novel Disease Detection Using Ensembles with Regularized Disagreement. In: Sudre, C.H., et al. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis. UNSURE PIPPI 2021 2021. Lecture Notes in Computer Science(), vol 12959. Springer, Cham. https://doi.org/10.1007/978-3-030-87735-4_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-87735-4_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87734-7
Online ISBN: 978-3-030-87735-4
eBook Packages: Computer ScienceComputer Science (R0)
