Skip to main content
Springer Nature Link
Log in

Analysing semi-supervised learning for image classification using compact networks in the biomedical context

  • Application of soft computing
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

The development of mobile and on the edge applications that embed deep convolutional neural models has the potential to revolutionise healthcare. However, most deep learning models require computational resources that are not available in smartphones or edge devices; an issue that can be faced by means of compact models that require less resources than standard deep learning models. The problem with such models is that they are, at least usually, less accurate than bigger models. We propose to address the accuracy limitation of compact networks with the application of semi-supervised learning techniques. In particular, we perform a thorough comparison of self-training methods, consistency regularisation techniques, and quantization techniques. We present a thorough analysis for the results obtained by combining 11 compact networks and 6 semi-supervised processes when applied to 10 biomedical datasets. Namely, combining semi-supervised methods and compact networks, we can create compact models that are not only as accurate as standard size models, but also faster and lighter. In addition, we have developed a Python library to facilitate the combination of compact networks and semi-supervised learning methods to tackle image classification tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

Enquiries about data availability should be directed to the authors.

Notes

  1. https://pytorch.org/docs/stable/quantization.html.

  2. https://github.com/oguiza/fastai_extensions.

  3. https://github.com/phanav/fixmatch-fastai.

References

  • Arredondo-Santoyo M, Domínguez C, Heras J et al (2019) Automatic characterisation of dye decolourisation in fungal strains using expert, traditional, and deep features. Soft Comput 23:12799–12812. https://doi.org/10.1007/s00500-019-03832-8

    Article  Google Scholar 

  • Azizi S, Mustafa B, Ryan F, Beaver Z, Freyberg J, Deaton J, Loh A, Karthikesalingam A, Kornblith S, Chen T et al (2021) Big self-supervised models advance medical image classification. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3478–3488

  • Berthelot D et al (2019) Mixmatch: a holistic approach to semi-supervised learning. In: 33rd international conference on neural information processing systems (NEURIPS’19). Curran Associates Inc, pp 5050–5060

  • Bucila C, Caruana R, Niculescu-Mizil A (2006) Model compression: making big, slow models practical. In: 12th international conference on knowledge discovery and data mining, KDD’06, pp. 535–541

  • Cai H, Gan C, Wang T, Zhang Z, Han S (2020) Once for all: train one network and specialize it for efficient deployment. In: International conference on learning representations. https://openreview.net/forum?id=HylxE1HKwS

  • Chaturvedi SS, Gupta K, Prasad PS (2020) Skin lesion analyser: an efficient seven-way multi-class skin cancer classification using mobilenet. In: AMLTA 2020: advanced machine learning technologies and applications. Advances in intelligent systems and computing, vol 1141. Springer, pp 165–176. https://doi.org/10.1007/978-981-15-3383-9_15

  • Chen S, Xiao N, Shi X, Yang Y, Tan H, Tian J, Quan Y (2023) Colormedgan: a semantic colorization framework for medical images. Appl Sci 13(5):3168

    Article  Google Scholar 

  • Chen T, Kornblith S, Swersky K, Norouzi M, Hinton GE (2020) Big self-supervised models are strong semi-supervised learners. Adv Neural Inf Process Syst 33:22243–22255

    Google Scholar 

  • Codella NCF, Gutman D, Celebi ME et al (2017) Skin lesion analysis toward melanoma detection: a challenge at the international symposium on biomedical imaging (ISBI) 2016. In: Conference on computer vision and pattern recognition, CVPR’17

  • Cohen J (1969) Statistical power analysis for the behavioral sciences. Academic Press, Cambridge

    Google Scholar 

  • Cubuk ED, et al (2020) Randaugment: Practical automated data augmentation with a reduced search space. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 3008–3017. IEEE . https://doi.org/10.1109/CVPRW50498.2020.00359

  • Deng J et al (2009) ImageNet: a large-scale hierarchical image database. In: IEEE computer society conference on computer vision and pattern recognition (CVPR09)

  • Garcia S et al (2010) Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Inf Sci 180:2044–2064

    Article  Google Scholar 

  • Giselsson TM, Jorgensen RN, Jensen PK et al (2017) A public image database for benchmark of plant seedling classification algorithms. arXiv:1711.05458

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  • Hinton G (2018) Deep learning—a technology with the potential to transform health care. JAMA 320(11):1101–1102. https://doi.org/10.1001/jama.2018.11100

  • Hinton G et al (2015) Distilling the knowledge in a neural network

  • Ho TKK, Gwak J (2020) Utilizing knowledge distillation in deep learning for classification of chest x-ray abnormalities. IEEE Access 8:160749–160761. https://doi.org/10.1109/ACCESS.2020.3020802

    Article  Google Scholar 

  • Howard J, Gugger S (2020) Fastai: a layered api for deep learning. Information 11:108. https://doi.org/10.3390/info11020108

    Article  Google Scholar 

  • Huang R, Noble JA, Namburete AIL (2018) Omni-supervised learning: Scaling up to large unlabelled medical datasets. In: medical image computing and computer assisted intervention, MICCAI’18. Springer, pp 572–580

  • Iandola FN et al (2016) Squeezenet: alexnet-level accuracy with 50x fewer parameters and \(<\)0.5mb model size

  • Inés A, Domínguez C, Heras J, Mata E, Pascual V (2021) Biomedical image classification made easier thanks to transfer and semi-supervised learning. Comput Methods Programs Biomed 198:105782

    Article  Google Scholar 

  • Irvin J, Rajpurkar P, Ko M et al (2019) Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Thirty-third AAAI conference on artificial intelligence, AAAI’19, vol 33, pp 590–597

  • Jiang H, Diao Z, Shi T, Zhou Y, Wang F, Hu W, Zhu X, Luo S, Tong G, Yao YD (2023) A review of deep learning-based multiple-lesion recognition from medical images: classification, detection and segmentation. Comput Biol Med 106726

  • Jiang N, Luo J, Lin J, Chen W, Zhao T (2023) Lightweight semi-supervised network for single image rain removal. Pattern Recogn 137:109277

    Article  Google Scholar 

  • Kaggle (2019a) Aptos 2019 blindness detection. https://www.kaggle.com/c/aptos2019-blindness-detection

  • Kaggle (2019b) Open sprayer images. https://www.kaggle.com/gavinarmstrong/open-sprayer-images

  • Kermany DS, Goldbaum M, Cai W et al (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5):1122-1131.e9. https://doi.org/10.1016/j.cell.2018.02.010

    Article  Google Scholar 

  • Krishnamoorthi R (2018) Quantizing deep convolutional networks for efficient inference: a whitepaper. arXiv:1806.08342

  • Kumar J, Ye P, Doermann D (2014) Structural similarity for document image classification and retrieval. Pattern Recogn Lett 43:119–126. https://doi.org/10.1016/j.patrec.2013.10.030

    Article  Google Scholar 

  • Li F, Song D, Chen H et al (2020) Development and clinical deployment of a smartphone-based visual field deep learning system for glaucoma detection. npj Digit Med 3:120. https://doi.org/10.1038/s41746-020-00329-9

    Article  Google Scholar 

  • Li G et al (2020) Soft-label anonymous gastric x-ray image distillation. In: 2020 IEEE international conference on image processing (ICIP). IEEE, pp 305–309. https://doi.org/10.1109/ICIP40778.2020.9191357

  • Liu Z, Mao H, Wu CY, Feichtenhofer C, Darrell T, Xie S (2022) A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  • Luo L, Xue D, Feng X (2020) Automatic diabetic retinopathy grading via self-knowledge distillation. Electronics 9(9):1337. https://doi.org/10.3390/electronics9091337

    Article  Google Scholar 

  • Marcolino MS, Oliveira JAQ, D’Agostino M et al (2018) The impact of mhealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 6(1):e23. https://doi.org/10.2196/mhealth.8873

    Article  Google Scholar 

  • Mehta S, Rastegari M (2022) Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. In: International conference on learning representations. https://openreview.net/forum?id=vh-0sUt8HlG

  • Meijering E (2020) A bird’s-eye view of deep learning in bioimage analysis. Comput Struct Biotechnol J 18(1):2312–2325. https://doi.org/10.1016/j.csbj.2020.08.003

    Article  Google Scholar 

  • Paszke A et al (2022) Pytorch: an imperative style, high-performance deep learning library. In: Wallach H, Larochelle H, Beygelzimer A, d’ Alché-Buc F, Fox E, Garnett R (eds) Advances in neural information processing systems, vol 32. Curran Associates, Inc., pp 8024–8035

  • Pogorelov K, Randel KR, Griwodz C et al (2017) Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In: 8th ACM on multimedia systems conference, MMSys’17. ACM, pp 164–169. https://doi.org/10.1145/3083187.3083212

  • Qayyum A, Tahir A, Butt MA, Luke A, Abbas HT, Qadir J, Arshad K, Assaleh K, Imran MA, Abbasi QH (2023) Dental caries detection using a semi-supervised learning approach. Sci Rep 13(1):749

    Article  Google Scholar 

  • Qin J, Wu J, Li M, Xiao X, Zheng M, Wang X (2022) Multi-granularity distillation scheme towards lightweight semi-supervised semantic segmentation. In: European conference on computer vision. Springer, pp 481–498

  • Razavian AS, Azizpour H, Sullivan J et al (2014) CNN features off-the-shelf: an astounding baseline for recognition. In: IEEE conference on computer vision and pattern recognition workshops, CVPRW’14. IEEE, pp 512–519

  • Sajjadi M et al (2016) Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In: 30th International conference on neural information processing systems (NIPS’16). Curran Associates Inc, pp 1171–1179

  • Sandler M, Howard A, Zhu M et al (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp 4510–4520. https://doi.org/10.1109/CVPR.2018.00474

  • Sheskin D (2011) Handbook of parametric and nonparametric statistical procedures. CRC Press, London

    Google Scholar 

  • Shuvo MMH (2022) Edge ai: leveraging the full potential of deep learning. In: Recent innovations in artificial intelligence and smart applications, pp 27–46. Springer

  • Shuvo MMH, Islam SK, Cheng J, Morshed BI (2023) Efficient acceleration of deep learning inference on resource-constrained edge devices: a review. Proc IEEE 111(1):42–91. https://doi.org/10.1109/JPROC.2022.3226481

    Article  Google Scholar 

  • Simard P, Steinkraus D, Platt JC (2003) Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of the 12th international conference on document analysis and recognition (ICDAR’03), vol 2, pp 958–964

  • Smith L (2017) Cyclical learning rates for training neural networks. In: IEEE Winter conference on applications of computer vision. WACV’17, pp 464–472. https://doi.org/10.1109/WACV.2017.58

  • Sohn K et al (2020) Fixmatch: simplifying semi-supervised learning with consistency and confidence. In: 34th international conference on neural information processing systems (NEURIPS’20). Curran Associates Inc

  • Suriyal S, Druzgalski C, Gautam K (2018) Mobile assisted diabetic retinopathy detection using deep neural network. In: 2018 global medical engineering physics exchanges/pan American Health Care Exchanges (GMEPE/PAHCE). IEEE, pp 1–4. https://doi.org/10.1109/GMEPE-PAHCE.2018.8400760

  • Tan M, Le QV (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In: International conference on machine learning, vol 97. ICML 2019, pp 6105–6114

  • Tan M, Le QV (2019) Mixconv: mixed depth wise convolutional kernels. In: 30th British machine vision conference 2019 (BMVC 2019), pp. 1–13

  • Tan M et al (2019) Mnasnet: platform-aware neural architecture search for mobile. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp 2815–2823. https://doi.org/10.1109/CVPR.2019.00293

  • Triguero I et al (2015) Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowl Inf Syst 42:245–284. https://doi.org/10.1007/s10115-013-0706-y

    Article  Google Scholar 

  • Tschandl P, Rosendahl C, Kittler H (2018) The HAM10000 dataset: a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data. https://doi.org/10.1038/sdata.2018.161

  • Upadhyay AK, Bhandari AK (2023) Semi-supervised modified-unet for lung infection image segmentation. IEEE Trans Radiat Plasma Med Sci. https://doi.org/10.1109/TRPMS.2023.3272209

    Article  Google Scholar 

  • Wiedemann S et al (2020) Deepcabac: a universal compression algorithm for deep neural networks. IEEE J Sel Top Signal Process 14(4):700–714. https://doi.org/10.1109/JSTSP.2020.2969554

    Article  Google Scholar 

  • Wu B, Dai X, Zhang P et al (2019) Fbnet: hardware-aware efficient convnet design via differentiable neural architecture search. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp 10726–10734. https://doi.org/10.1109/CVPR.2019.01099

  • Zhang X et al (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp 6848–6856. https://doi.org/10.1109/CVPR.2018.00716

  • Zhu X, Goldberg AB (2009) Introduction to semi-supervised learning, synthesis lectures on artificial intelligence and machine learning. Morgan & Claypool Publishers, San Rafael

    Google Scholar 

Download references

Funding

This work was partially supported by Ministerio de Ciencia e Innovación [PID2020-115225RB-I00 / AEI / 10.13039/501100011033], and FPU Grant 16/06903 of the Spanish MEC.

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Sciences, University of La Rioja, Logroño, Spain

    Adrián Inés, César Domínguez, Jónathan Heras, Eloy Mata & Vico Pascual

  2. Department of Biomedical Engineering, School of Biomedical Engineering Imaging Sciences, King’s College London, London, UK

    Andrés Díaz-Pinto

Authors
  1. Adrián Inés
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Andrés Díaz-Pinto
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. César Domínguez
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Jónathan Heras
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Eloy Mata
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Vico Pascual
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

AI: methodology, software, validation, investigation, and writing—original draft. AD-P: conceptualization, and writing—review and editing. CD: supervision, funding acquisition, validation, formal analysis, and writing—review and editing. JH: supervision, validation, formal analysis, and writing—review & editing. EM supervision and writing—review and editing. VP: supervision, funding acquisition, and writing—review and editing.

Corresponding author

Correspondence to Adrián Inés.

Ethics declarations

Conflict of interest

None declared.

Ethical approval

Not applicable.

Informed consent

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Inés, A., Díaz-Pinto, A., Domínguez, C. et al. Analysing semi-supervised learning for image classification using compact networks in the biomedical context. Soft Comput 28, 8931–8943 (2024). https://doi.org/10.1007/s00500-023-09109-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-09109-5

Keywords