Skip to main content

Advertisement

Springer Nature Link
Log in

A Collaborative Training Using Crowdsourcing and Neural Networks on Small and Difficult Image Classification Datasets

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Although machine learning has recently achieved performance that exceeds human capacity in prediction, humans still have an advantage in difficult tasks when the number of training samples is small or when human knowledge is required to identify features that are included in samples. However, if specialized knowledge is required, the number of humans that can perform those tasks is limited. In this study, we effectively use crowdsourcing to incorporate domain knowledge in neural network trainings; specifically, we decide feature values by asking crowdsourcing workers to answer easy questions prepared based on dictionary. We evaluated this method on a single type of task that is intuitive and relevant for non-specialists, which is binary classification of dog image datasets with similar breeds, and found that using crowdsourcing tended to improve the performance of machine-learning models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Albarqouni S, Baur C, Achilles F, Belagiannis V, Demirci S, Navab N. AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans Med Imaging. 2016;35(5):1313–21.

    Article  Google Scholar 

  2. Amazon Mechanical Turk. https://www.mturk.com/. Accessed 1 Oct 2020.

  3. Bejnordi BE, Veta M, van Diest PJ, van Ginneken B, Karssemeijer N, Litjens G, van der Laak JA, Hermsen M, Manson Q, Balkenhol M, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. J Am Med Assoc (JAMA). 2017;318(22):2199–210.

    Article  Google Scholar 

  4. Chatzimilioudis G, Konstantinidis A, Laoudias C, Zeinalipour-Yazti D. Crowdsourcing with smartphones. IEEE Internet Comput. 2012;16(5):36–44.

    Article  Google Scholar 

  5. Cubuk ED, Zoph B, Mane D, Vasudevan V, Le QV. AutoAugment: learning augmentation policies from data. arXiv preprint: arXiv: 1805.09501. 2018.

  6. de Herrera AGS, Foncubierta-Rodríguez A, Markonis D, Schaer R, Müller H. Crowdsourcing for medical image classification. Swiss Med Inform. 2014;30:13.

    Google Scholar 

  7. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. An image is worth 16x16 words: transformers for image recognition at scale. In: Proceedings of the international conference on learning representations (ICLR). 2021.

  8. Duan X, Tajima K. Improving multiclass classification in crowdsourcing using hierarchical schemes. In: Proceedings of the world wide web conference (WWW). 2019. p. 13–17.

  9. Feng S, Zhou H, Dong H. Using deep neural network with small dataset to predict material defects. Mater Des. 2019;162(15):300–10.

    Article  Google Scholar 

  10. Gal Y, Islam R, Ghahramani Z. Deep bayesian active learning with image data. Proc Int Conf Mach Learn (ICML). 2017;70:1183–92.

    Google Scholar 

  11. Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In: Proceedings of the international conference on artificial intelligence and statistics (AISTATS). 2011. p. 315–323.

  12. He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE international conference on computer vision (ICCV). 2015. p. 1026–1034.

  13. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2016. p. 770–778.

  14. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. 2017.

  15. Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2018. p. 7132–7141.

  16. Kazai G, Kamps J, Koolen M, Milic-Frayling N. Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking. In: Proceedings of the international ACM SIGIR conference on research and development in information retrieval (SIGIR). 2011. p. 205–214.

  17. Khosla A, Jayadevaprakash N, Yao B, Fei-Fei L. Novel dataset for fine-grained image categorization. In: The first workshop on fine-grained visual categorization (FGVC). 2011.

  18. Kingma DP, Ba J. Adam: a method for stochastic optimization. In: Proceedings of the international conference on learning representations (ICLR). 2015.

  19. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems (NIPS). 2012. p. 1097–1105.

  20. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998;86(11):2278–324.

    Article  Google Scholar 

  21. Legg P, Smith J, Downing A. Visual analytics for collaborative human-machine confidence in human-centric active learning tasks. In: Human-centric computing and information sciences. vol. 9. 2019.

  22. Lu P, Li B, Shama S, King I, Chan JH. Regularizing the loss layer of CNNs for facial expression recognition using crowdsourced labels. In: Proceedings of the Asia Pacific symposium on intelligent and evolutionary systems (IES). 2017.

  23. Ørting S, Doyle A, Hilten van MHA et al. A survey of crowdsourcing in medical image analysis. arXiv preprint: arXiv:1902.09159. 2019.

  24. Pan SJ, Yang Q. A survey on transfer learning. IEEE Trans Knowl Data Eng. 2010;22(10):1345–59.

    Article  Google Scholar 

  25. Perez L, Wang J. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint: arXiv:1712.04621. 2017.

  26. Pratt LY. Discriminability-based transfer between neural networks. Adv Neural Inf Process Syst (NIPS). 1993;5:204–11.

    Google Scholar 

  27. Rogez G, Schmid C. MoCap-guided data augmentation for 3D pose estimation in the wild. In: Advances in neural information processing systems (NIPS). 2016.

  28. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L. ImageNet large scale visual recognition challenge. Int J Comput Vis. 2015;115:211–52.

    Article  MathSciNet  Google Scholar 

  29. Salamon J, Bello JP. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Process Lett. 2017;24(3):279–83.

    Article  Google Scholar 

  30. Saralioglu E, Gungor O. Crowdsourcing-based application to solve the problem of insufficient training data in deep learning-based classification of satellite images. Geocarto Int. 2021; 1–20.

  31. Sener O, Savarese S. Active learning for convolutional neural networks: a core-set approach. In: Proceedings of the international conference on learning representations (ICLR). 2018.

  32. Settles B. Active learning literature survey. In: Computer science technical report 1648, University of Wisconsin-Madison. 2009.

  33. Sharma M, Saha O, Sriraman A, Hebbalaguppe R, Vig L, Karande S. Crowdsourcing for chromosome segmentation and deep classification. In: The IEEE conference on computer vision and pattern recognition (CVPR) workshops. 2017. p. 34–41.

  34. Tan M, Le QV. EfficientNet: RethinkingModel scaling for convolutional neural networks. In: Proceedings of the international conference on machine learning (ICML). 2019.

  35. Tao D, Cheng J, Yu Z, Yue K, Wang L. Domain-weighted majority voting for crowdsourcing. IEEE Trans Neural Netw Learn Syst. 2018;30(1):163–74.

    Article  MathSciNet  Google Scholar 

  36. Verma V, Lamb A, Beckham C, Couville A, Mitli-agkas I, Bengio Y. Manifold mixup: better representations by interpolating hidden states. In: Proceedings of the international conference on machine learning (ICML). 2019.

  37. Wang J, Yang Y, Mao J, Huang Z, Huang C, Xu W. CNN-RNN: a unified framework for multi-label image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2016. p. 2285–2294.

  38. Wang K, Zhang D, Li Y, Zhang R, Lin L. Cost-effective active learning for deep image classification. IEEE Trans Circuits Syst Video Technol. 2016;27(12):2591–600.

    Article  Google Scholar 

  39. Wang X, Pham H, Dai Z, Neubig G. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP). 2018.

  40. Wang Y, Gu M, Ma J, Jin Q. DNN-DP: differential privacy enabled deep neural network learning framework for sensitive crowdsourcing data. IEEE Trans Comput Soc Syst. 2020;7(1):215–24.

    Article  Google Scholar 

  41. Xu Z, Liu Y, Yen N, Mei L, Luo X, Wei X, Hu C. Crowdsourcing based description of urban emergency events using social media big data. IEEE Trans Cloud Comput. 2016;8(2):387–97.

    Article  Google Scholar 

  42. Yun S, Han D, Oh SJ, Chun S, Choe J, Yoo Y. CutMix: regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE international conference on computer vision (ICCV). 2019. p. 6023–6032.

  43. Zamir AR, Sax A, Shen W, Guibas L, Malik J, Savarese S. Taskonomy: disentangling task transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2018. p. 3712–3722.

  44. Zanzotto FM. Human-in-the-loop artificial intelligence. J Artif Intell Res. 2019;64:243–52.

    Article  MathSciNet  Google Scholar 

  45. Zhang H, Cisse M, Dauphin YN, Lopez-Paz D. mixup: beyond empirical risk minimization. In: Proceedings of international conference on learning representations (ICLR). 2018.

  46. Zhang Z, Jing J, Wang X, Choo K-KR, Gupta BB. A crowdsourcing method for online social networks security assessment based on human-centric computing. Human-centric Comput Inf Sci. 2020;10.

Download references

Acknowledgements

This study is based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO), Japan.

Author information

Authors and Affiliations

  1. Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan

    Tomoumi Takase

Authors
  1. Tomoumi Takase
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Tomoumi Takase.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Takase, T. A Collaborative Training Using Crowdsourcing and Neural Networks on Small and Difficult Image Classification Datasets. SN COMPUT. SCI. 3, 178 (2022). https://doi.org/10.1007/s42979-022-01076-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01076-2

Keywords