Skip to main content
SpringerLink
Log in

Supervised contrastive learning with corrected labels for noisy label learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Deep neural networks have achieved significant success in the artificial intelligence community and various downstream tasks. They encode images or texts into dense feature representations and are supervised by a large amount of labeled data. Due to the expensiveness of high-quality labeled data, a huge number of easy-to-access instances are collected to conduct supervised learning. However, they have not been annotated by experts and thus can contain numerous noisy instances, which will degrade the performance. To learn robust feature representations despite misleading noisy labels, we employ supervised contrastive learning to directly perform supervision in the hidden space, rather than in the prediction space like the prevalent cross-entropy loss function. However, cutting-edge noisy label learning methods with supervised contrastive learning always discard the data considered to be noisy, and thus cannot tolerate high-ratio noisy datasets. Therefore, we propose a novel training strategy named Supervised Contrastive Learning with Corrected Labels (Scl \(^2\)) to defend against the attack of noisy labels. Scl \(^2\) corrects the noisy labels with an empirical small-loss assumption and conducts supervised contrastive learning using these corrected data. Specifically, we employ the generated soft labels as supervisory information to facilitate our implementation of supervised contrastive learning. This expansion of contrastive learning ensures the integrity of the supervisory information while effectively enhancing the learning process. In addition, samples sharing the same soft labels are treated as positive sample pairs, while those with different soft labels are considered to be negative sample pairs. With this strategy, the representations from neural networks keep the local discrimination in one mini-batch. Besides, we also employ a prototype contrastive learning technique to ensure global discrimination. Our Scl \(^2\) has demonstrated excellent performance on numerous benchmark datasets, showcasing its effectiveness in various standardized evaluation scenarios. Additionally, our model has proven to be highly valuable when applied to real-world noisy datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability and access

Source code for the experiments is available at https://github.com/ChenyangLu922/SCL2.git.

References

  1. Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput 29(9):2352–2449

    Article  MathSciNet  MATH  Google Scholar 

  2. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  3. Khosla P, Teterwak P, Wang C et al (2020) Supervised contrastive learning. Adv Neural Inf Process Syst 33:18661–18673

    Google Scholar 

  4. Chen T, Kornblith S, Norouzi M et al (2020) A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning. PMLR, pp 1597–1607

  5. Li C, Li X, Ouyang J (2021b) Semi-supervised text classification with balanced deep representation distributions. In: Annual Meeting of the Association for Computational Linguistics. pp 5044–5053

  6. Feng S, Wang B, Yang Z et al (2022) Aspect-based sentiment analysis with attention-assisted graph and variational sentence representation. Knowl-Based Syst 258:109975

  7. Ouyang J, Wang Y, Li X et al (2022) Weakly-supervised text classification with Wasserstein Barycenters regularization. In: International Joint Conference on Artificial Intelligence. pp 3373–3379

  8. Li X, Wang B, Wang Y et al (2023b) Weakly supervised prototype topic model with discriminative seed words: modifying the category prior by self-exploring supervised signals. Soft Comput 27(9):5397–5410

  9. Raza K, Singh NK (2021) A tour of unsupervised deep learning for medical image analysis. Current Medical Imaging 17(9):1059–1077

    Google Scholar 

  10. Paolacci G, Chandler J, Ipeirotis PG (2010) Running experiments on Amazon mechanical Turk. Judgm Decis Mak 5(5):411–419

    Article  Google Scholar 

  11. Arpit D, Jastrzebski S, Ballas N et al (2017) A closer look at memorization in deep networks. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, pp 233–242

  12. Zhang C, Bengio S, Hardt M et al (2021a) Understanding deep learning (still) requires rethinking generalization. Commun ACM 64(3):107–115

  13. Zhang Z, Sabuncu MR (2018) Generalized cross entropy loss for training deep neural networks with noisy labels. pp 8792–8802

  14. Tanaka D, Ikami D, Yamasaki T et al (2018) Joint optimization framework for learning with noisy labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 5552–5560

  15. Ma X, Huang H, Wang Y et al (2020) Normalized loss functions for deep learning with noisy labels. In: International Conference on Machine Learning. pp 6543–6553

  16. Han B, Yao Q, Yu X et al (2018) Co-teaching: robust training of deep neural networks with extremely noisy labels. Adv Neural Inf Process Syst 31:8536–8546

    Google Scholar 

  17. Tan C, Xia J, Wu L et al (2021) Co-learning: Learning from noisy labels with self-supervision. In: Proceedings of the 29th ACM International Conference on Multimedia. pp 1405–1413

  18. Yu X, Han B, Yao J et al (2019) How does disagreement help generalization against label corruption? In: International Conference on Machine Learning. PMLR, pp 7164–7173

  19. Wei H, Feng L, Chen X et al (2020) Combating noisy labels by agreement: A joint training method with co-regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 13723–13732

  20. Xiao T, Xia T, Yang Y et al (2015) Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 2691–2699

  21. Wang Y, Liu W, Ma X et al (2018) Iterative learning with open-set noisy labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 8688–8696

  22. Yao Y, Sun Z, Zhang C et al (2021) JO-SRC: a contrastive approach for combating noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 5192–5201

  23. Li S, Xia X, Ge S et al (2022a) Selective-supervised contrastive learning with noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 316–325

  24. Huang B, Lin Y, Xu C (2022) Contrastive label correction for noisy label learning. Inf Sci 611:173–184

    Article  Google Scholar 

  25. He K, Fan H, Wu Y et al (2020) Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference On Computer Vision and Pattern Recognition. pp 9729–9738

  26. Wang B, Ding L, Zhong Q et al (2022) A contrastive cross-channel data augmentation framework for aspect-based sentiment analysis. In: International Conference on Computational Linguistics. pp 6691–6704

  27. Ortego D, Arazo E, Albert P et al (2021) Multi-objective interpolation training for robustness to label noise. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 6606–6615

  28. Natarajan N, Dhillon IS, Ravikumar P et al (2013) Learning with noisy labels. Adv Neural Inf Proces Syst 26

  29. Patrini G, Rozza A, Menon AK et al (2017) Making deep neural networks robust to label noise: a loss correction approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 2233–2241

  30. Xia X, Liu T, Wang N et al (2019) Are anchor points really indispensable in label-noise learning? pp 6835–6846

  31. Goldberger J, Ben-Reuven E (2017) Training deep neural-networks using a noise adaptation layer. In: International Conference on Learning Representations

  32. Ghosh A, Kumar H, Sastry PS (2017) Robust loss functions under label noise for deep neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp 1919–1925

  33. Wang Y, Ma X, Chen Z et al (2019) Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 322–330

  34. Jiang L, Zhou Z, Leung T et al (2018) Mentornet: learning data-driven curriculum for very deep neural networks on corrupted labels. In: International Conference on Machine Learning. PMLR, pp 2309–2318

  35. Chen P, Liao B, Chen G et al (2019) Understanding and utilizing deep neural networks trained with noisy labels. In: International Conference on Machine Learning. PMLR, pp 1062–1070

  36. Li X, Jiang Y, Li C et al (2023a) Learning with partial labels from semi-supervised perspective. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp 8666–8674

  37. Li X, Wang Y (2020) Recovering accurate labeling information from partially valid data for effective multi-label learning. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020. pp 1373–1380

  38. Li C, Li X, Feng L et al (2021a) Who is your right mixup partner in positive and unlabeled learning. In: International Conference on Learning Representations

  39. Yu K, Lin TR, Ma H et al (2021) A multi-stage semi-supervised learning approach for intelligent fault diagnosis of rolling bearing using data augmentation and metric learning. Mech Syst Signal Process 146:107043

    Article  Google Scholar 

  40. Li X, Lu P, Hu L et al (2022b) A novel self-learning semi-supervised deep learning network to detect fake news on social media. Multimed Tools Appl 81(14):19341–19349

  41. Yan Y, Xu Z, Tsang IW et al (2016) Robust semi-supervised learning through label aggregation. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp 2244–2250

  42. Nguyen DT, Mummadi CK, Ngo T et al (2020) Self: learning to filter noisy labels with self-ensembling. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

  43. Ding Y, Wang L, Fan D et al (2018) A semi-supervised two-stage approach to learning from noisy labels. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, pp 1215–1224

  44. Li J, Socher R, Hoi SCH (2020) Dividemix: learning with noisy labels as semi-supervised learning. In: 8th International Conference on Learning Representations, ICLR 2020,Addis Ababa, Ethiopia, April 26–30, 2020

  45. Berthelot D, Carlini N, Goodfellow IJ et al (2019) Mixmatch: a holistic approach to semi-supervised learning. Adv Neural Inf Proces Syst 32:5050–5060

    Google Scholar 

  46. Grill J, Strub F, Altché F et al (2020) Bootstrap your own latent-a new approach to self-supervised learning. Adv Neural Inf Process Syst 33:21271–21284

    Google Scholar 

  47. Chen X, He K (2021) Exploring simple Siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 15750–15758

  48. Yi L, Liu S, She Q et al (2022) On learning contrastive representations for learning with noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 16661–16670

  49. Reed SE, Lee H, Anguelov D et al (2015) Training deep neural networks on noisy labels with bootstrapping. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Workshop Track Proceedings

  50. Lu Y, He W (2022) SELC: self-ensemble label correction improves learning with noisy labels. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23–29 July 2022. pp 3278–3284

  51. Krizhevsky A (2009) Learning multiple layers of features from tiny images. Master’s thesis, University of Tront

  52. Song H, Kim M, Lee J (2019) Selfie: Refurbishing unclean samples for robust deep learning. In: International Conference on Machine Learning. PMLR, pp 5907–5915

  53. Liu S, Niles-Weed J, Razavian N et al (2020) Early-learning regularization prevents memorization of noisy labels. Adv Neural Inf Proces Syst 33:20331–20342

  54. Chen P, Ye J, Chen G et al (2021) Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp 11442–11450

  55. Zhang Y, Zheng S, Wu P et al (2021b) Learning with feature-dependent label noise: a progressive approach. In: International Conference on Learning Representations

Download references

Funding

This work was partially supported by the National Natural Science Foundation of China (NSFC) [No.62006094, No.62276113] and the Project Funded by China Postdoctoral Science Foundation (No.2022M721321).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Jilin University, Changchun, China

    Jihong Ouyang, Chenyang Lu, Bing Wang & Changchun Li

  2. Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, China

    Jihong Ouyang, Chenyang Lu, Bing Wang & Changchun Li

Authors
  1. Jihong Ouyang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenyang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Changchun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jihong Ouyang: Resources, Project administration, Supervision. Chenyang Lu: Methodology, Software, Investigation, Writing - Original Draft, Data Curation, Validation. Bing Wang: Writing - Review & Editing, Visualization. Changchun Li: Conceptualization, Supervision.

Corresponding author

Correspondence to Changchun Li.

Ethics declarations

Ethical and informed consent for data used

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. We the undersigned declare that this manuscript entitled “Supervised Contrastive Learning with Corrected Labels for Noisy Label Learning” is original, has not been published before, and is not currently being considered for publication elsewhere.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ouyang, J., Lu, C., Wang, B. et al. Supervised contrastive learning with corrected labels for noisy label learning. Appl Intell 53, 29378–29392 (2023). https://doi.org/10.1007/s10489-023-05018-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05018-0

Keywords

Search

Navigation