Abstract
Deep learning based classification of biomedical images requires expensive manual annotation by experts. Incomplete-supervision approaches including active learning, pre-training, and semi-supervised learning have thus been developed to increase classification performance with a limited number of annotated images. In practice, a combination of these approaches is often used to reach the desired performance for biomedical images.
Most of these approaches are designed for natural images, which differ fundamentally from biomedical images in terms of color, contrast, image complexity, and class imbalance. In addition, it is not always clear which combination to use in practical cases.
We, therefore, analyzed the performance of combining seven active learning, three pre-training, and two semi-supervised methods on four exemplary biomedical image datasets covering various imaging modalities and resolutions. The results showed that the ImageNet (pre-training) in combination with pseudo-labeling (semi-supervised learning) dominates the best performing combinations, while no particular active learning algorithm prevailed. For three out of four datasets, this combination reached over 90% of the fully supervised results by only adding 25% of labeled data. An ablation study also showed that pre-training and semi-supervised learning contributed up to 25% increase in F1-score in each cycle. In contrast, active learning contributed less than 5% increase in each cycle.
Based on these results, we suggest employing the correct combination of pre-training and semi-supervised learning can be more efficient than active learning for biomedical image classification with limited annotated images. We believe that our study is an important step towards annotation-efficient model training for biomedical classification challenges.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Data Availability Statement
All scripts and how to access and process the data can be found here: https://github.com/marrlab/Med-AL-SSL.
References
Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., Liu, C.: A survey on deep transfer learning. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 270–279. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01424-7_27
Chen, T., Kornblith, S., Swersky, K., et al.: Big self-supervised models are strong semi-supervised learners (2020). http://arxiv.org/abs/2006.10029
Raghu, M., Zhang, C., Kleinberg, J., et al.: Transfusion: understanding transfer learning for medical imaging. In: Wallach, H., Larochelle, H., Beygelzimer, A., et al. (eds.) Advances in Neural Information Processing Systems 32, pp. 3347–3357. Curran Associates Inc (2019)
Matek, C., Schwarz, S., Marr, C., Spiekermann, K.: A single-cell morphological dataset of leukocytes from AML patients and non-malignant controls [Data set]. Cancer Imaging Arch. (2019). https://doi.org/10.7937/tcia.2019.36f5o9ld
Wang, X., Peng, Y., Lu, L., et al.: ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases (2017). http://arxiv.org/abs/1705.02315
Blasi, T., Hennig, H., Summers, H.D., et al.: Label-free cell cycle analysis for high-throughput imaging flow cytometry. Nat. Commun. 7, 10256 (2016)
Jing, L., Tian, Y.: Self-supervised visual feature learning with deep neural networks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(11), 4037–4058 (2020)
Joshi, A.J., Porikli, F., Papanikolopoulos, N.: Multi-class active learning for image classification. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2372–2379 (2009)
Chen, T., Kornblith, S., Norouzi, M., et al.: A simple framework for contrastive learning of visual representations (2020). http://arxiv.org/abs/2002.05709
Zhou, Z.-H.: A brief introduction to weakly supervised learning. Natl. Sci. Rev. 5, 44–53 (2017)
Sadafi, A., et al.: Multiclass deep active learning for detecting red blood cell subtypes in brightfield microscopy. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 685–693. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_76
Combalia, M., Codella, N.C.F., Rotemberg, V., et al.: BCN20000: dermoscopic lesions in the wild (2019). http://arxiv.org/abs/1908.02288
Holub, A., Perona, P., Burl, M.C.: Entropy-based active learning for object recognition. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8 (2008)
Ren, P., Xiao, Y., Chang, X., et al.: A Survey of Deep Active Learning (2020). http://arxiv.org/abs/2009.00236
Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results (2017). http://arxiv.org/abs/1703.01780
Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5, 180161 (2018)
van Engelen, J.E., Hoos, H.H.: A survey on semi-supervised learning. Mach. Learn. 109(2), 373–440 (2019). https://doi.org/10.1007/s10994-019-05855-6
Esteva, A., Kuprel, B., Novoa, R.A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature (2017)
Newell, A., Deng, J.: How Useful is Self-Supervised Pretraining for Visual Tasks? (2020). http://arxiv.org/abs/2003.14323
Ducoffe, M., Precioso, F.: QBDC: query by dropout committee for training deep supervised architecture (2015). http://arxiv.org/abs/1511.06412
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)
Sohn, K., Berthelot, D., Li, C.-L., et al.: FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020). http://arxiv.org/abs/2001.07685
Wei, K., Iyer, R., Bilmes, J.: Submodularity in data subset selection and active learning. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning, pp. 1954–1963 PMLR, Lille, France (2015)
Settles, B.: Active Learning Literature Survey, University of Wisconsin-Madison Department of Computer Sciences (2009)
Sagheer, A., Kotb, M.: Unsupervised pre-training of a deep LSTM-based stacked autoencoder for multivariate time series forecasting problems. Sci. Rep. 9, 19038 (2019)
Culotta, A., McCallum, A.: Reducing labeling effort for structured prediction tasks. In: Veloso, M.M., Kambhampati, S. (eds.) Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, July 9–13, 2005, Pittsburgh, Pennsylvania, USA, pp. 746–751. AAAI Press / The MIT Press (2005)
Sener, O., Savarese, S.: Active Learning for Convolutional Neural Networks: A Core-Set Approach (2017). http://arxiv.org/abs/1708.00489
Rajpurkar, P., Irvin, J., Zhu, K., et al.: CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning (2017). http://arxiv.org/abs/1711.05225
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Ash, J.T., Zhang, C., Krishnamurthy, A., et al.: Deep batch active learning by diverse, uncertain gradient lower bounds (2019). http://arxiv.org/abs/1906.03671
Eulenberg, P., Köhler, N., Blasi, T., et al.: Reconstructing cell cycle and disease progression using deep learning. Nat. Commun. 8, 463 (2017)
van Oord, A,D., Li, Y., Vinyals, O.: Representation Learning with Contrastive Predictive Coding (2018). http://arxiv.org/abs/1807.03748
Yoo, D., Kweon, I.S.: Learning loss for active learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 93–102 (2019)
Matek, C., Schwarz, S., Spiekermann, K., et al.: Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks. Nat. Mach. Intell. 1, 538–544 (2019). https://doi.org/10.1038/s42256-019-0101-9
Clark, K., et al.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26(6), 1045–1057 (2013). https://doi.org/10.1007/s10278-013-9622-7
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., et al.: GLISTER: generalization based data subset selection for efficient and robust learning (2020). http://arxiv.org/abs/2012.10630
APTOS 2019 Blindness Detection. https://www.kaggle.com/c/aptos2019-blindness-detection/
Zhou, J., Sun, S.: Improved margin sampling for active learning. In: Li, S., Liu, C., Wang, Y. (eds.) CCPR 2014. CCIS, vol. 483, pp. 120–129. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-45646-0_13
Codella, N.C.F., Gutman, D., Celebi, E.M., et al.: Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (ISIC) (2017). http://arxiv.org/abs/1710.05006
Gal, Y., Islam, R., Ghahramani, Z.: Deep bayesian active learning with image data (2017). http://arxiv.org/abs/1703.02910
Ericsson, L., Gouk, H., Hospedales, T.M.: How well do self-supervised models transfer? (2020). http://arxiv.org/abs/2011.13377
Taher, M.R.H., Haghighi, F., Feng, R., et al.: A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis (2021). https://doi.org/10.1007/978-3-030-87722-4_1, http://arxiv.org/abs/2108.05930
Acknowledgment
We thank Björn Menze, Tingying Peng, Christian Matek, Melanie Schulz, Rudolf Matthias Hehr, Lea Schuh, Valerio Lupperger, and Ario Sadafi (Munich) for discussions and for contributing their ideas.
Funding
SSB has received funding by F. Hoffmann-la Roche LTD and supported by the Helmholtz Association under the joint research school “Munich School for Data Science - MUDS”. CM has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (Grant agreement No. 866411).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shetab Boushehri, S., Qasim, A.B., Waibel, D., Schmich, F., Marr, C. (2022). Systematic Comparison of Incomplete-Supervision Approaches for Biomedical Image Classification. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13529. Springer, Cham. https://doi.org/10.1007/978-3-031-15919-0_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-15919-0_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15918-3
Online ISBN: 978-3-031-15919-0
eBook Packages: Computer ScienceComputer Science (R0)