Skip to main content
Springer Nature Link
Log in

Data fusing and joint training for learning with noisy labels

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

It is well known that deep learning depends on a large amount of clean data. Because of high annotation cost, various methods have been devoted to annotating the data automatically. However, a larger number of the noisy labels are generated in the datasets, which is a challenging problem. In this paper, we propose a new method for selecting training data accurately. Specifically, our approach fits a mixture model to the per-sample loss of the raw label and the predicted label, and the mixture model is utilized to dynamically divide the training set into a correctly labeled set, a correctly predicted set, and a wrong set. Then, a network is trained with these sets in the supervised learning manner. Due to the confirmation bias problem, we train the two networks alternately, and each network establishes the data division to teach the other network. When optimizing network parameters, the labels of the samples fuse respectively by the probabilities from the mixture model. Experiments on CIFAR-10, CIFAR-100 and Clothing1M demonstrate that this method is the same or superior to the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Yan Y, Rosales R, Fung G, Subramanian R, Dy J. Learning from multiple annotators with varying expertise. Machine Learning, 2014, 95(3): 291–327

    Article  MathSciNet  Google Scholar 

  2. Yu X, Liu T, Gong M, Tao D. Learning with biased complementary labels. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 69–85

  3. Blum A, Kalai A, Wasserman H. Noise-tolerant learning, the parity problem, and the statistical query model. Journal of the ACM, 2003, 50(4): 506–519

    Article  MathSciNet  Google Scholar 

  4. Tanno R, Saeedi A, Sankaranarayanan S, Alexander D C, Silberman N. Learning from noisy labels by regularized estimation of annotator confusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, 11236–11245

  5. Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. Understanding deep learning requires rethinking generalization. In: Proceedings of the 5th International Conference on Learning Representations (ICLR). 2017

  6. Goldberger J, Ben-Reuven E. Training deep neural-networks using a noise adaptation layer. In: Proceedings of the ICLR. 2017

  7. Liu T, Tao D. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(3): 447–461

    Article  MathSciNet  Google Scholar 

  8. Patrini G, Rozza A, Menon A K, Nock R, Qu L. Making deep neural networks robust to label noise: a loss correction approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017, 2233–2241

  9. Reed S E, Lee H, Anguelov D, Szegedy C, Erhan D, Rabinovich A. Training deep neural networks on noisy labels with bootstrapping. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR). 2015

  10. Tanaka D, Ikami D, Yamasaki T, Aizawa K. Joint optimization framework for learning with noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018, 5552–5560

  11. Arazo E, Ortego D, Albert P, O’Connor N E, McGuinness K. Unsupervised label noise modeling and loss correction. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 312–321

  12. Zhang H, Cissé M, Dauphin Y N, Lopez-Paz D. mixup: beyond empirical risk minimization. In: Proceedings of the 6th International Conference on Learning Representations (ICLR). 2018

  13. Ren M, Zeng W, Yang B, Urtasun R. Learning to reweight examples for robust deep learning. In: Proceedings of the 35th International Conference on Machine Learning (ICML). 2018, 4331–4340

  14. Jiang L, Zhou Z, Leung T, Li L, Fei-Fei L. MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels. In: Proceedings of the 35th International Conference on Machine Learning (ICML). 2018, 2309–2318

  15. Han B, Yao Q, Yu X, Niu G, Xu M, Hu W, Tsang I W, Sugiyama M. Co-teaching: robust training of deep neural networks with extremely noisy labels. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS). 2018, 8536–8546

  16. Yu X, Han B, Yao J, Niu G, Tsang I W, Sugiyama M. How does disagreement help generalization against label corruption? In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 7164–7173

  17. Wei H, Feng L, Chen X, An B. Combating noisy labels by agreement: a joint training method with co-regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020, 13723–13732

  18. Li J, Socher R, Hoi S C H. DivideMix: learning with noisy labels as semi-supervised learning. In: Proceedings of the 8th International Conference on Learning Representations. 2020

  19. Li Y, Yang J, Song Y, Cao L, Luo J, Li L. Learning from noisy labels with distillation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017, 1928–1936

  20. Xiao T, Xia T, Yang Y, Huang C, Wang X. Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015, 2691–2699

  21. Vahdat A. Toward robustness against label noise in training deep discriminative neural networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS). 2017, 5601–5610

  22. Veit A, Alldrin N, Chechik G, Krasin I, Gupta A, Belongie S J. Learning from noisy large-scale datasets with minimal supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017, 6575–6583

  23. Lee K H, He X, Zhang L, Yang L. CleanNet: transfer learning for scalable image classifier training with label noise. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018, 5447–5456

  24. Yi K, Wu J. Probabilistic end-to-end noise correction for learning with noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, 7010–7018

  25. Ma X, Wang Y, Houle M E, Zhou S, Erfani S M, Xia S, Wijewickrema S N R, Bailey J. Dimensionality-driven learning with noisy labels. In: Proceedings of the 35th International Conference on Machine Learning (ICML). 2018, 3361–3370

  26. Hendrycks D, Mazeika M, Wilson D, Gimpel K. Using trusted data to train deep networks on labels corrupted by severe noise. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS). 2018, 10477–10486

  27. Thulasidasan S, Bhattacharya T, Bilmes J A, Chennupati G, Mohd-Yusof J. Combating label noise in deep learning using abstention. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 6234–6243

  28. Shen Y, Sanghavi S. Learning with bad training data via iterative trimmed loss minimization. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 5739–5748

  29. Ghosh A, Kumar H, Sastry P S. Robust loss functions under label noise for deep neural networks. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI). 2017, 1919–1925

  30. Wang Y, Ma X, Chen Z, Luo Y, Yi J, Bailey J. Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2019, 322–330

  31. Ding Y, Wang L, Fan D, Gong B. A semi-supervised two-stage approach to learning from noisy labels. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV). 2018, 1215–1224

  32. Kong K, Lee J, Kwak Y, Kang M, Kim S G, Song W J. Recycling: semi-supervised learning with noisy labels in deep neural networks. IEEE Access, 2019, 7: 66998–67005

    Article  Google Scholar 

  33. Berthelot D, Carlini N, Goodfellow I J, Oliver A, Papernot N, Raffel C. MixMatch: a holistic approach to semi-supervised learning. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS). 2019, 454

  34. Arpit D, Jastrzebski S, Ballas N, Krueger D, Bengio E, Kanwal M S, Maharaj T, Fischer A, Courville A C, Bengio Y, Lacoste-Julien S. A closer look at memorization in deep networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML). 2017, 233–242

  35. Chen P, Liao B, Chen G, Zhang S. Understanding and utilizing deep neural networks trained with noisy labels. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 1062–1070

  36. Permuter H, Francos J, Jermyn I. A study of Gaussian mixture models of color and texture features for image classification and segmentation. Pattern Recognition, 2006, 39(4): 695–706

    Article  Google Scholar 

  37. Tarvainen A, Valpola H. Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS). 2017, 1195–1204

  38. Krizhevsky A. Learning multiple layers of features from tiny images. University of Toronto, Dissertation, 2009

  39. Li J, Wong Y, Zhao Q, Kankanhalli M S. Learning to learn from noisy labeled data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, 5046–5054

  40. Wang Y, Liu W, Ma X, Bailey J, Zha H, Song L, Xia S T. Iterative learning with open-set noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018, 8688–8696

  41. He K, Zhang X, Ren S, Sun J. Identity mappings in deep residual networks. In: Proceedings of the 14th European Conference on Computer Vision (ECCV). 2016, 630–645

Download references

Acknowledgements

This work was supported by SRC-Open Project of Research Center of Security Video and Image Processing Engineering Technology of Guizhou ([2020]001]), Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRS20) and National Natural Science Foundation of China (Grant No. 61973334).

Author information

Authors and Affiliations

  1. College of Electrical Engineering and Control Science, Nanjing Tech University, Nanjing, 211816, China

    Yi Wei, Mei Xue & Pengxiang Xu

  2. Beijing Seetatech Technology Co., Ltd, Beijing, 100029, China

    Xin Liu

Authors
  1. Yi Wei
    View author publications

    Search author on:PubMed Google Scholar

  2. Mei Xue
    View author publications

    Search author on:PubMed Google Scholar

  3. Xin Liu
    View author publications

    Search author on:PubMed Google Scholar

  4. Pengxiang Xu
    View author publications

    Search author on:PubMed Google Scholar

Corresponding authors

Correspondence to Mei Xue or Xin Liu.

Additional information

Yi Wei received his Master’s degree from Nanjing Tech University, China in 2021. His research interest is deep learning and robustness to noisy labels.

Mei Xue received her PhD degree in pattern recognition and intelligent system from Southeast University, China in 2008. Currently, she is a professor and the Master supervisor at Nanjing Tech University, China. Her current research interests include pattern recognition, machine vision and image processing.

Xin Liu received his BS degree in software engineering from Chongqing University, China in June 2011, and Ph.D. degree in computer science from Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), China in 2017. He is now the CEO of SeetaTech, which is an AI startup in China. His research interests include face recognition, image processing and deep learning.

Pengxiang Xu received his Master’s degree from Nanjing Tech University, China in 2021. His research interest is face manipulation detection.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, Y., Xue, M., Liu, X. et al. Data fusing and joint training for learning with noisy labels. Front. Comput. Sci. 16, 166338 (2022). https://doi.org/10.1007/s11704-021-1208-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-021-1208-9

Keywords