Skip to main content
SpringerLink
Log in

Ensemble of multi-task deep convolutional neural networks using transfer learning for fruit freshness classification

  • 1200: Machine Vision Theory and Applications for Cyber Physical Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automatic classification of fruit freshness plays an important role in the agriculture industry. In this work, we propose an ensemble model that combines the bottleneck features of two multi-task deep convolutional neural networks with different architectures (ResNet-50 and ResNet-101). In our proposed multi-tasking framework, there are two classification branches: a binary classifier to distinguish between fresh and rotten fruits, and a multi-class label classifier to identify the kind of fruit. Since the features (e.g., color, texture, and shape) of rotten fruits are different from each other depending on the kind of fruit, the input of the first branch is combined with the kind of fruit information from the second branch to classify the fruit freshness more accurately. Transfer learning technique has been applied during the model training since transfer learning has been shown to be effective transfer learning has been shown to be effective in many applications in which training data for the target problem are limited. To evaluate our proposed model, we use simple images from the existing dataset and real-world images crawled from the web, both representing fresh and rotten fruits for different fruit categories as our dataset. Our proposed model achieved average accuracies of 98.50% and 97.43% for freshness classification and fruit classification, respectively, demonstrating that our transfer learning-based ensemble model outperforms other transfer learning-based models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Akçay S, Kundegorski ME, Devereux M, Breckon TP (2016) Transfer learning using convolutional neural networks for object classification within x-ray baggage security imagery. In: 2016 IEEE International Conference on Image Processing (ICIP), IEEE, pp 1057–1061

  2. Akhtar MS, Chauhan DS, Ghosal D, Poria S, Ekbal A, Bhattacharyya P (2019) Multi-task learning for multi-modal emotion recognition and sentiment analysis. arXiv preprint. arXiv: 190505812

  3. Al-Twairesh N, Al-Negheimish H (2019) Surface and deep features ensemble for sentiment analysis of arabic tweets. IEEE Access 7:84122–84131

  4. Albawi S, Mohammed TA, Al-Zawi S (2017) Understanding of a convolutional neural network. In: 2017 International Conference on Engineering and Technology (ICET), IEEE 1–6

  5. Altaheri H, Alsulaiman M, Muhammad G (2019) Date fruit classification for robotic harvesting in a natural environment using deep learning. IEEE Access 7:117115–117133

  6. Ananthanarayana T, Ptucha R, Kelly SC (2020) Deep learning based fruit freshness classification and detection with cmos image sensors and edge processors. Electron Imaging 12:172–1

    Google Scholar 

  7. Baltruschat IM, Nickisch H, Grass M, Knopp T, Saalbach A (2019) Comparison of deep learning approaches for multi-label chest x-ray classification. Sci Rep 9(1):1–10

    Article  Google Scholar 

  8. Calisto MB, Lai-Yuen SK (2020a) Adaen-net: An ensemble of adaptive 2d–3d fully convolutional networks for medical image segmentation. Neural Network

  9. Calisto MGB, Lai-Yuen SK (2020b) Self-adaptive 2d-3d ensemble of fully convolutional networks for medical image segmentation. In: Medical Imaging 2020: Image Processing, International Society for Optics and Photonics 11313:113131W

  10. Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75

    Article  MathSciNet  Google Scholar 

  11. Christodoulidis S, Anthimopoulos M, Ebner L, Christe A, Mougiakakou S (2016) Multisource transfer learning with convolutional neural networks for lung pattern analysis. IEEE J Biomed Health Inform 21(1):76–84

    Article  Google Scholar 

  12. Corchs S, Fersini E, Gasparini F (2019) Ensemble learning on visual and textual data for social image emotion classification. Int J Mach Learn Cybern 10(8):2057–2070

    Article  Google Scholar 

  13. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint. arXiv: 181004805

  14. Enciso-Aragón CJ, Pachón-Suescún CG, Jimenez-Moreno R (2018) Quality control system by means of cnn and fuzzy systems. Int J Appl Eng Res 13(16):12846–12853

  15. Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. JIPS 10(3):443–458

    Google Scholar 

  16. Gómez-Sanchis J, Martín-Guerrero JD, Soria-Olivas E, Martínez-Sober M, Magdalena-Benedito R, Blasco J (2012) Detecting rottenness caused by penicillium genus fungi in citrus fruits using machine learning techniques. Expert Systems with Applications 39(1):780–785

    Article  Google Scholar 

  17. Goyal M, Goyal R, Lall B (2019) Learning activation functions: A new paradigm of understanding neural networks. arXiv preprint. arXiv: 190609529

  18. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc IEEE Conf Comput Vis Pattern Recognit 770–778

  19. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint. arXiv: 170404861

  20. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proc IEEE Conf Comput Vis Pattern Recognit 4700–4708

  21. Kang J, Gwak J (2019) Ensemble of instance segmentation models for polyp segmentation in colonoscopy images. IEEE Access 7:26440–26447

  22. Kang J, Gwak J (2020) Ensemble learning of lightweight deep learning models using knowledge distillation for image classification. Mathematics 8(10):1652

    Article  Google Scholar 

  23. Kang J, Choi H, Lee H (2019) Deep recurrent convolutional networks for inferring user interests from social media. J Intell Inf Syst 52(1):191–209

    Article  Google Scholar 

  24. Kang J, Ullah Z, Gwak J (2021) Mri-based brain tumor classification using ensemble of deep features and machine learning classifiers. Sensors 21(6):2222

    Article  Google Scholar 

  25. Karakaya D, Ulucan O, Turkan M (2020) A comparative analysis on fruit freshness classification. In: 2019 Innovations in Intelligent Systems and Applications Conference (ASYU), IEEE 1–4

  26. Khoong WH (2020) Busu-net: An ensemble u-net framework for medical image segmentation. arXiv preprint. arXiv: 200301581

  27. Krizhevsky A (2014) One weird trick for parallelizing convolutional neural networks. arXiv preprint. arXiv: 14045997

  28. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Adv Neural Inf Proces Syst 1097–1105

  29. Lee Hy, Hu Ty, Jing H, Chang YF, Tsao Y, Kao YC, Pao TL (2013) Ensemble of machine learning and acoustic segment model techniques for speech emotion and autism spectrum disorders recognition. In: INTERSPEECH 215–219

  30. López M, Valdivia A, Martínez-Cámara E, Luzón MV, Herrera F (2019) E2sam: evolutionary ensemble of sentiment analysis methods for domain adaptation. Inf Sci 480:273–286

    Article  Google Scholar 

  31. Ma N, Zhang X, Zheng HT, Sun J (2018) Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV) 116–131

  32. Majumder N, Poria S, Peng H, Chhaya N, Cambria E, Gelbukh A (2019) Sentiment and sarcasm classification with multitask learning. IEEE Intell Syst 34(3):38–43

    Article  Google Scholar 

  33. Minaee S, Azimi E, Abdolrashidi A (2019) Deep-sentiment: Sentiment analysis using ensemble of cnn and bi-lstm models. arXiv preprint. arXiv: 190404206

  34. Nguyen HH, Fang F, Yamagishi J, Echizen I (2019) Multi-task learning for detecting and segmenting manipulated facial images and videos. arXiv preprint. arXiv: 190606876

  35. Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  36. Phan HT, Tran VC, Nguyen NT, Hwang D (2020) Improving the performance of sentiment analysis of tweets containing fuzzy sentiment using the feature ensemble model. IEEE Access 8:14630–14641

  37. Polikar R (2006) Ensemble based systems in decision making. IEEE Circuits Syst Mag 6(3):21–45

    Article  Google Scholar 

  38. Prasomphan S, Doungwichain S (2017) Detecting human emotion via speech recognition by using ensemble classification model. In: International Conference on Big Data Technologies and Applications, Springer 66–73

  39. Ren Y, Zhang L, Suganthan PN (2016) Ensemble classification and regression-recent developments, applications and future directions. IEEE Comput Intell Mag 11(1):41–53

    Article  Google Scholar 

  40. Rocha A, Hauagge DC, Wainer J, Goldenstein S (2010) Automatic fruit and vegetable classification from images. Comput Electron Agric 70(1):96–104

    Article  Google Scholar 

  41. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proc IEEE Conf Comput Vis Pattern Recognit 4510–4520

  42. Semary NA, Tharwat A, Elhariri E, Hassanien AE (2015) Fruit-based tomato grading system using features fusion and support vector machine. In: Intelligent Systems’ 2014, Springer 401–410

  43. Sifre L, Mallat S (2014) Rigid-motion scattering for texture classification. arXiv preprint. arXiv: 14031687

  44. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489

  45. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint. arXiv: 14091556

  46. Singh S, Singh NP (2019) Machine learning-based classification of good and rotten apple. In: Recent trends in communication, computing, and electronics, Springer 377–386

  47. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proc IEEE Conf Comput Vis Pattern Recognit 1–9

  48. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proc IEEE Conf Comput Vis Pattern Recognit 2818–2826

  49. Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J (2016) Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging 35(5):1299–1312

    Article  Google Scholar 

  50. Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) Mnasnet: Platform-aware neural architecture search for mobile. In: Proc IEEE Conf Comput Vis Pattern Recognit 2820–2828

  51. Tao F, Liu G, Zhao Q (2018) An ensemble framework of voice-based emotion recognition system. In: 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), IEEE 1–6

  52. Wang L, Li A, Tian X (2013) Detection of fruit skin defects using machine vision system. In: 2013 Sixth International Conference on Business Intelligence and Financial Engineering, IEEE 44–48

  53. Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn Comput 9(5):597–610

    Article  Google Scholar 

  54. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proc IEEE Conf Comput Vis Pattern Recognit 1492–1500

  55. Yang H, Mei S, Song K, Tao B, Yin Z (2017) Transfer-learning-based online mura defect classification. IEEE Trans Semicond Manuf 31(1):116–123

    Article  Google Scholar 

  56. Yu J, Gwak J, Lee S, Jeon M (2015) An incremental learning approach for restricted boltzmann machines. 2015 International Conference on Control. Automation and Information Sciences (ICCAIS), IEEE, pp 113–117

    Google Scholar 

  57. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proc IEEE Conf Comput Vis Pattern Recognit 6848–6856

  58. Zhang Y, Wu L (2012) Classification of fruits using computer vision and a multiclass support vector machine. Sensors 12(9):12489–12505

  59. Zhang YD, Dong Z, Chen X, Jia W, Du S, Muhammad K, Wang SH (2019) Image based fruit category classification by 13-layer deep convolutional neural network and data augmentation. Multimed Tools Appl 78(3):3613–3632

    Article  Google Scholar 

  60. Zvarevashe K, Olugbara O (2020) Ensemble learning of hybrid acoustic features for speech emotion recognition. Algorithms 13(3):70

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (Grant No. NRF-2020R1I1A3074141), the Brain Research Program through the NRF funded by the Ministry of Science, ICT and Future Planning (Grant No. NRF-2019M3C7A1020406), and Regional Innovation Strategy (RIS) through the NRF funded by the Ministry of Education.

Author information

Authors and Affiliations

  1. Department of Software, Korea National University of Transportation, Chungju, 27469, Korea

    Jaeyong Kang & Jeonghwan Gwak

  2. Department of AI Robotics Engineering, Korea National University of Transportation, Chungju, 27469, Korea

    Jeonghwan Gwak

  3. Department of Biomedical Engineering, Korea National University of Transportation, Chungju, 27469, Korea

    Jeonghwan Gwak

  4. Department of IT Energy Convergence (BK21 FOUR), Korea National University of Transportation, Chungju, 27469, Korea

    Jeonghwan Gwak

Authors
  1. Jaeyong Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeonghwan Gwak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeonghwan Gwak.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kang, J., Gwak, J. Ensemble of multi-task deep convolutional neural networks using transfer learning for fruit freshness classification. Multimed Tools Appl 81, 22355–22377 (2022). https://doi.org/10.1007/s11042-021-11282-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11282-4

Keywords

Search

Navigation