Skip to main content
SpringerLink
Log in

Deep learning based automatic grading of bi-colored apples using multispectral images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Grading of apple fruits involves their inspection, assessment and sorting by quality. Machine vision has been the industry’s choice as it is fast, reliable and tireless. Recently deep learning has brought revolutionary advances in computer vision and machine learning. Accordingly this study presents a deep learning based quality grading solution for apple fruits. A 2D convolutional neural network is trained on multispectral images of bi-colored apples to realize two-category and multi-category grading. For the multi-category grading, the performance of a novel cascaded CNNs based solution is further investigated. Experimental results show that the proposed deep learning solution achieves highly accurate and fast grading performance outperforming the state-of-the-art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Alam MN, Ullah I, Al-Absi AA (2020) Deep learning-based apple defect detection with residual squeezenet. In: International conference on smart computing and cyber security: strategic foresight, security challenges and innovation. Springer, pp 127–134

  2. Anonymous (2004) Commission regulation (ec) no 85/2004 of 15 january 2004 on marketing standards for apples. Off J Eur Union L 13:3–18

    Google Scholar 

  3. Ariana D, Guyer DE, Shrestha B (2006) Integrating multispectral reflectance and fluorescence imaging for defect detection on apples. Comput Electron Agric 50(2):148–161

    Article  Google Scholar 

  4. Bhatt AK, Pant D (2015) Automatic apple grading model development based on back propagation neural network and machine vision, and its performance evaluation. Ai Soc 30(1):45–56

    Article  Google Scholar 

  5. Cheng X, Tao Y, Chen YR, Luo Y (2003) Nir/mir dual-sensor machine vision system for online apple stem-end/calyx recognition. Trans ASAE 46:551–558

    Article  Google Scholar 

  6. Crowe TG, Delwiche MJ (1996) Real-time defect detection in fruit - part i: Design concepts and development of protoype hardware. Trans ASAE 39:2299–2308

    Article  Google Scholar 

  7. Cubero S, Aleixos N, Moltó E, Gómez-Sanchis J, Blasco J (2011) Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food Bioprocess Technol 4(4):487–504

    Article  Google Scholar 

  8. Davenel A, Guizard C, Labarre T, Sevila F (1988) Automatic detection of surface defects on fruit by using a vision system. J Agric Eng Res 41:1–9

    Article  Google Scholar 

  9. Diener RG, Mitchell JP, Rhoten ML (1970) Using an x-ray image scan to sort bruised apples. Agric Eng 51:356–361

    Google Scholar 

  10. ElMasry G, Wang N, Vigneault C (2009) Detecting chilling injury in Red Delicious apple using hyperspectral imaging and neural networks. Postharvest Biol Technol 52(1):1–8

    Article  Google Scholar 

  11. Elzebroek A, Wind K (2008) Guide to cultivated plants. Cabi Series, CABI

  12. Fan S, Li J, Zhang Y, Tian X, Wang Q, He X, Zhang C, Huang W (2020) On line detection of defective apples using computer vision system combined with deep learning methods. J Food Eng:110102

  13. Geoola F, Geoola F, Peiper UM (1994) A spectrophotometric method for detecting surface bruises on ‘golden delicious’ apples. J Agric Eng Res 58:47–51

    Article  Google Scholar 

  14. Hu Z, Tang J, Zhang P, Jiang J (2020) Deep learning for the identification of bruised apples by fusing 3d deep features for apple grading systems. Mech Syst Signal Process 145:106922

    Article  Google Scholar 

  15. Ismail N, Malik OA (2021) Real-time visual inspection system for grading fruits using computer vision and deep learning techniques. Inf Process Agric

  16. Kavdir I, Guyer DE (2004) Comparison of artificial neural networks and statistical classifiers in apple sorting using textural features. Biosyst Eng 89:331–344

    Article  Google Scholar 

  17. Keresztes JC, Goodarzi M, Saeys W (2016) Real-time pixel based early apple bruise detection using short wave infrared hyperspectral imaging in combination with calibration and glare correction techniques. Food Control 66:215–226

    Article  Google Scholar 

  18. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. Cite arXiv:1412.6980 Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego 2015

  19. Kleynen O, Leemans V, Destain MF (2003) Selection of the most efficient wavelength bands for ‘jonagold’ apple sorting. Postharvest Biol Technol 30:221–232

    Article  Google Scholar 

  20. Kleynen O, Leemans V, Destain MF (2005) Development of a multi-spectral vision system for the detection of defects on apples. J Food Eng 69:41–49

    Article  Google Scholar 

  21. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  22. Kumar MP, Parkavi A (2020) Quality grading of the fruits and vegetables using image processing techniques and machine learning: a review. In: Advances in communication systems and networks. Springer, pp 477–486

  23. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436

    Article  Google Scholar 

  24. Leemans V, Destain MF (2004) A real-time grading method of apples based on features extracted from defects. J Food Eng 61:83–89

    Article  Google Scholar 

  25. Leemans V, Magein H, Destain MF (2002) On-line fruit grading according to their external quality using machine vision. Biosyst Eng 83:397–404

    Article  Google Scholar 

  26. Li Y, Feng X, Liu Y, Han X (2021) Apple quality identification and classification by computer vision based on deep learning. Scientific Reports

  27. Lorente D, Aleixos N, Gȯmez-Sanchis J, Cubero S, García-Navarrete OL, Blasco J (2012) Recent advances and applications of hyperspectral imaging for fruit and vegetable quality assessment. Food Bioprocess Technol 5(4):1121–1142

    Article  Google Scholar 

  28. Lu R (2003) Detection of bruises on apples using near-infrared hyperspectral imaging. Trans ASAE 46:523–530

    Google Scholar 

  29. Ma L, Bi S, Zhang C (2019) Apple grading system based on near infrared spectroscopy and evidential classification forest. In: International conference on advanced mechatronic systems, ICAMechS, IEEE Computer Society, vol 2019-August, pp 326–330

  30. Mehl PM, Chen YR, Kim MS, Chen DE (2004) Development of hyperspectral imaging technique for the detection of apple surface defects and contaminations. J Food Eng 61:67–81

    Article  Google Scholar 

  31. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Fürnkranz J, Joachims T (eds) ICML. Omni Press, pp 807–814

  32. Peng Y, Lu R (2008) Analysis of spatially resolved hyperspectral scattering images for assessing apple fruit firmness and soluble solids content. Postharvest Biology Technol 48(1):52–62

    Article  Google Scholar 

  33. Rehman TU, Mahmud MS, Chang YK, Jin J, Shin J (2019) Current and future applications of statistical machine learning algorithms for agricultural machine vision systems. Comput Electron Agric 156:585–605

    Article  Google Scholar 

  34. Saranya N, Srinivasan K, Pravin Kumar SK, Rukkumani V, Ramya R (2019) Fruit classification using traditional machine learning and deep learning approach. In: Smys S, Tavares J, Balas VIA (eds) Computational vision and bio-inspired computing. ICCVBIC 2019, advances in intelligent systems and computing, vol 1108. Springer, Cham, pp 79–89

  35. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626

  36. Shahin MA, Tollner EW, McClendon RW, Arabnia HR (2002) Apple classification based on surface bruises using image processing and neural networks. Trans ASAE 45:1619–1627

    Google Scholar 

  37. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  38. Sofu MM, Er O, Kayacan M, Cetiṡli B (2016) Design of an automatic apple sorting system using machine vision. Comput Electron Agric 127:395–405

    Article  Google Scholar 

  39. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: A simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    MathSciNet  MATH  Google Scholar 

  40. Sun DW (2010) Hyperspectral imaging for food quality analysis and control. Elsevier Science, London

    Google Scholar 

  41. Tan W, Zhao C, Wu H (2016) Intelligent alerting for fruit-melon lesion image based on momentum deep learning. Multimed Tools Appl 75 (24):16741–16761

    Article  Google Scholar 

  42. Tang Y, Gao S, Zhuang J, Hou C, He Y, Chu X, Miao A, Luo S (2020) Apple bruise grading using piecewise nonlinear curve fitting for hyperspectral imaging data. IEEE Access 8:147494–147506

    Article  Google Scholar 

  43. Throop JA, Aneshansley DJ, Anger WC, Peterson DL (2005) Quality evaluation of apples based on surface defects: development of an automated inspection system. Postharvest Biol Technol 36:281–290

    Article  Google Scholar 

  44. Toylan H, Kuscu H (2014) A real-time apple grading system using multicolor space. Sci World J 2014:2356–6140

    Article  Google Scholar 

  45. Unay D, Gosselin B (2007) Stem and calyx recognition on ‘jonagold’ apples by pattern recognition. J Food Eng 78:597–605

    Article  Google Scholar 

  46. Unay D, Gosselin B, Kleynen O, Leemans V, Destain MF, Debeir O (2011) Automatic grading of bi-colored apples by multispectral machine vision. Comput Electron Agric 75(1):204–212

    Article  Google Scholar 

  47. Unay D, Destain MF, Gosselin B, Kleynen O, Leemans V (2018) The CAPA apple quality grading multi-spectral image database. Dataset on Zenodo

  48. Upchurch BL, Affeldt HA, Hruschka WR, Throop JA (1991) Optical detection of bruises and early frost damage on apples. Trans ASAE 34:1004–1009

    Article  Google Scholar 

  49. Valdez P (2020) Apple defect detection using deep learning based object detection for better post harvest handling. arXiv:200506089

  50. Wen Z, Tao Y (1999) Building a rule-based machine-vision system for defect inspection on apple sorting and packing lines. Expert Syst Appl 16:307–313

    Article  Google Scholar 

  51. Wu A, Zhu J, Ren T (2020) Detection of apple defect using laser-induced light backscattering imaging and convolutional neural network. Comput Electr Eng 81:106454

    Article  Google Scholar 

  52. Xiao-bo Z, Jie-wen Z, Yanxiao L, Holmes M (2010) In-line detection of apple defects using three color cameras system. Comput Electron Agric 70(1):129–134

    Article  Google Scholar 

  53. Zhang B, Huang W, Li J, Zhao C, Fan S, Wu J, Liu C (2014) Principles, developments and applications of computer vision for external quality inspection of fruits and vegetables: a review. Food Res Int 62:326–343

    Article  Google Scholar 

  54. Zhou L, Zhang C, Liu F, Qiu Z, He Y (2019) Application of deep learning in food: a review. Compr Rev Food Sci Food Saf 18(6):1793–1811

    Article  Google Scholar 

  55. Zhu L, Spachos P, Pensini E, Plataniotis KN (2021) Deep learning and machine vision for food processing: a survey. Curr Res Food Sci 4:233–249

    Article  Google Scholar 

Download references

Acknowledgements

This work is funded by the General Directorate of Technology, Research and Energy of the Walloon Region of Belgium with Convention No 9813783.

Author information

Authors and Affiliations

  1. Electrical and Electronics Engineering, Izmir Demokrasi University, Izmir, Turkey

    Devrim Unay

Authors
  1. Devrim Unay
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Devrim Unay.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Unay, D. Deep learning based automatic grading of bi-colored apples using multispectral images. Multimed Tools Appl 81, 38237–38252 (2022). https://doi.org/10.1007/s11042-022-12230-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12230-6

Keywords

Search

Navigation