Skip to main content
SpringerLink
Log in

Combining an information-maximization-based attention mechanism and illumination invariance theory for the recognition of green apples in natural scenes

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Accurate recognition of green fruit targets is one of the key technologies for fruit growth monitoring and yield estimation. To solve the problem of fruit misidentification due to the similarity between fruit skin and leaf colors, a progressive detection method of green apples in natural environments was proposed. Image enhancement based on fuzzy set theory was carried out to make the fruit targets more salient in the whole image. Then, the fruit areas were roughly determined by the attention-based information maximization (AIM) algorithm, and the recognized apple regions were cropped according to the adaptive pixel-extending method to remove the background information. After that, accurate segmentation of fruit targets was accomplished by fusing the illumination-invariant image and R-component of the cropped image. To evaluate the performance of this method, it was compared with the illumination invariance theory-based algorithm, mean shift algorithm, K-means clustering algorithm, manifold ranking algorithm and GrabCut algorithm. The test was conducted using 200 green apple images under different growth statuses. Experimental results showed that the segmentation rate of the proposed method was 86.91%, which was 3.26%, 6.35%, 16.43%, 3.08% and 4.7% higher than those of the other five methods, respectively. The false positive rate and false negative rate were 0.88% and 10.53%, which gained an advantage over those of the other five segmentation algorithms. The localization error was 3.65%. In conclusion, the proposed method can accurately segment green fruit targets, which can lay the foundation for intelligent management of fruits over the entire growing season.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Bansal R, Bansal R, Lee WS, Lee WS, Satish S, Satish S (2013) Green citrus detection using fast Fourier transform (FFT) leakage. Precis Agric 14(1):59–70. https://doi.org/10.1007/s11119-012-9292-3

    Article  Google Scholar 

  2. Boini A, Manfrini L, Bortolotti G, Corelli-Grappadelli L, Morandi B (2019) Monitoring fruit daily growth indicates the onset of mild drought stress in apple. Sci Hortic 256:108520. https://doi.org/10.1016/j.scienta.2019.05.047

    Article  Google Scholar 

  3. Bruce NDB, Tsotsos JK (2005) Saliency based on information maximization. In: Proceedings of the 18th International Conference on Neural Information Processing Systems, pp 155–162

  4. Craciun S, Craciun S, Kirchgessner R, Kirchgessner R, George AD, George AD, Lam H, Lam H, Principe JC, Principe JC (2018) A real-time, power-efficient architecture for mean-shift image segmentation. J Real-Time Image Pr 14(2):379–394. https://doi.org/10.1007/s11554-014-0459-1

    Article  Google Scholar 

  5. Dubey SR, Dixit P, Singh N, Gupta JP (2013) Infected fruit part detection using K-means clustering segmentation technique. Int J Interact Multimed Artif Intell 2(2):65–72. https://doi.org/10.9781/ijimai.2013.229

    Article  Google Scholar 

  6. Dubey SR, Dubey SR, Singh SK, Singh SK, Singh RK, Singh RK (2015) A multi-channel based illumination compensation mechanism for brightness invariant image retrieval. Multimed Tools Appl 74(24):11223–11253. https://doi.org/10.1007/s11042-014-2226-5

    Article  Google Scholar 

  7. Finlayson GD, Hordley SD, Lu C, Drew MS (2006) On the removal of shadows from images. IEEE T Pattern Anal 28(1):59–68. https://doi.org/10.1109/TPAMI.2006.18

    Article  Google Scholar 

  8. Hassaballah M, Ghareeb A (2017) A framework for objective image quality measures based on intuitionistic fuzzy sets. Appl Soft Comput 57:48–59. https://doi.org/10.1016/j.asoc.2017.03.046

    Article  Google Scholar 

  9. He ZL, Xiong JT, Lin R, Zou X, Tang LY, Yang ZG, Liu Z, Song G (2017) A method of green litchi recognition in natural environment based on improved LDA classifier. Comput Electron Agric 140:159–167. https://doi.org/10.1016/j.compag.2017.05.029

  10. Huang X, Li G, Ma C, Yang S (2018) Green peach recognition based on improved discriminative regional feature integration algorithm in similar background. Transactions of the Chinese Society of Agricultural Engineering 34(23):142–148. https://doi.org/10.11975/j.issn.1002-6819.2018.23.017

    Article  Google Scholar 

  11. Kang F, Wang C, Li J, Zong Z (2018) A multiobjective piglet image segmentation method based on an improved noninteractive GrabCut algorithm. Adv Multimedia 2018:1083876–1083879. https://doi.org/10.1155/2018/1083876

    Article  Google Scholar 

  12. Lee T-W, Girolami M, Sejnowski TJ (1999) Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural Comput 11(2):417–441. https://doi.org/10.1162/089976699300016719

    Article  Google Scholar 

  13. Li H, Lee WS, Wang K (2016) Immature green citrus fruit detection and counting based on fast normalized cross correlation (FNCC) using natural outdoor colour images. Precis Agric 17(6):678–697. https://doi.org/10.1007/s11119-016-9443-z

    Article  Google Scholar 

  14. Li DH, Zhao H, Yu X (2019) Overlapping green apple recognition based on improved spectral. Spectrosc Spect Anal 39(9):2974–2981. https://doi.org/10.3964/j.issn.1000-0593(2019)09-2974-08

    Article  Google Scholar 

  15. Liu Z, Sun C, Bai X, Zhou F (2014) Infrared ship target image smoothing based on adaptive mean shift. In: Proceedings of Digital Image Computing: Techniques and Applications, pp 1–8. https://doi.org/10.1109/DICTA.2014.7008113

  16. Liu X, Jia W, Ruan C, Zhao D, Gu Y, Chen W (2018) The recognition of apple fruits in plastic bags based on block classification. Precis Agric 19(4):735–749. https://doi.org/10.1007/s11119-017-9553-2

  17. Lu X, Wang W, Ma C, Shen J, Shao L (2013) Porikli F see more, know more: unsupervised video object segmentation with co-attention Siamese networks. In: IEEE Conference on Computer Vision and Pattern Recognition. pp 3623–3632. https://doi.org/10.1109/CVPR.2019.00374

  18. Lu J, Lee WS, Gan H, Hu X (2018) Immature citrus fruit detection based on local binary pattern feature and hierarchical contour analysis. Biosyst Eng 171:78–90. https://doi.org/10.1016/j.biosystemseng.2018.04.009

    Article  Google Scholar 

  19. Lu X, Ma C, Ni B, Yang X, Reid I, Yang M-H (2018) Deep regression tracking with shrinkage loss. In: Computer Vision – ECCV 2018, pp 369–386. https://doi.org/10.1007/978-3-030-01264-9_22

    Article  Google Scholar 

  20. Lungu R, Lungu M, Grigorie LT (2013) ALSs with conventional and fuzzy controllers considering wind shear and gyro errors. J Aerospace Eng 26(4):794–813. https://doi.org/10.1061/(ASCE)AS.1943-5525.0000207

    Article  Google Scholar 

  21. Mitra S, Pal SK (2005) Fuzzy sets in pattern recognition and machine intelligence. Fuzzy Sets Syst 156(3):381–386. https://doi.org/10.1016/j.fss.2005.05.035

    Article  MathSciNet  Google Scholar 

  22. Mizushima A, Lu R (2013) An image segmentation method for apple sorting and grading using support vector machine and Otsu's method. Comput Electron Agric 94:29–37. https://doi.org/10.1016/j.compag.2013.02.009

    Article  Google Scholar 

  23. Okamoto H, Lee WS (2009) Green citrus detection using hyperspectral imaging. Comput Electron Agric 66(2):201–208. https://doi.org/10.1016/j.compag.2009.02.004

    Article  Google Scholar 

  24. Pham VH, Pham VH, Lee BR, Lee BR (2015) An image segmentation approach for fruit defect detection using k-means clustering and graph-based algorithm. Vietnam J Comput Sci 2(1):25–33. https://doi.org/10.1007/s40595-014-0028-3

    Article  Google Scholar 

  25. Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE T Pattern Anal 39(4):640–651. https://doi.org/10.1109/TPAMI.2016.2572683

    Article  Google Scholar 

  26. Skovsen S, Dyrmann M, Mortensen AK, Steen KA, Green O, Eriksen J, Gislum R, Jørgensen RN, Karstoft H (2017) Estimation of the botanical composition of clover-grass leys from RGB images using data simulation and fully convolutional neural networks. Sensors 17(12):2930. https://doi.org/10.3390/s17122930

    Article  Google Scholar 

  27. Song H, Qu W, Wang D, Yu X, He D (2014) Shadow removal method of apples based on illumination invariant image. Transactions of the Chinese Society of Agricultural Engineering 30(24):168–176. https://doi.org/10.3969/j.issn.1002-6819.2014.24.020

    Article  Google Scholar 

  28. Stajnko D, Lakota M, Hočevar M (2004) Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Comput Electron Agric 42(1):31–42. https://doi.org/10.1016/S0168-1699(03)00086-3

    Article  Google Scholar 

  29. Sun S, Wu Q, Jiao L, Long Y, He D, Song H (2018) Recognition of green apples based on fuzzy set theory and manifold ranking algorithm. Optik 165:395–407. https://doi.org/10.1016/j.ijleo.2018.03.085

    Article  Google Scholar 

  30. Wang D, Song H, Tie Z, Zhang W, He D (2016) Recognition and localization of occluded apples using K-means clustering algorithm and convex hull theory: a comparison. Multimed Tools Appl 75(6):3177–3198. https://doi.org/10.1007/s11042-014-2429-9

  31. Wang C, Lee WS, Zou X, Choi D, Gan H, Diamond J (2018) Detection and counting of immature green citrus fruit based on the local binary patterns (LBP) feature using illumination-normalized images. Precis Agric 19(6):1062–1083. https://doi.org/10.1007/s11119-018-9574-5

    Article  Google Scholar 

  32. Wang ZL, Walsh K, Koirala A (2019) Mango fruit load estimation using a video based MangoYOLO-Kalman filter-Hungarian algorithm method. Sensors 19(12):2742. https://doi.org/10.3390/s19122742

    Article  Google Scholar 

  33. Wang D, He D, Song H, Liu C, Xiong H (2019) Combining SUN-based visual attention model and saliency contour detection algorithm for apple image segmentation. Multimed Tools Appl 78(13):17391–17411. https://doi.org/10.1007/s11042-018-7106-y

    Article  Google Scholar 

  34. Wang W, Lu X, Shen J, Crandall D, Shao L (2019) Zero-shot video object segmentation via attentive graph neural networks. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp 9235–9244. https://doi.org/10.1109/ICCV.2019.00933

  35. Yang C, Zhang L, Lu H, Ruan X, Yang M-H saliency detection via graph-based manifold ranking. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 3166–3173. https://doi.org/10.1109/CVPR.2013.407

  36. Zadeh LA (1965) Fuzzy sets. Inf Control 8(3):338–353. https://doi.org/10.1016/S0019-9958(65)90241-X

    Article  MATH  Google Scholar 

  37. Zhao C, Lee WS, He D (2016) Immature green citrus detection based on colour feature and sum of absolute transformed difference (SATD) using colour images in the citrus grove. Comput Electron Agric 124:243–253. https://doi.org/10.1016/j.compag.2016.04.009

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the funding organizations for financial support. The authors would also like to appreciate all the authors cited in this article and anonymous referees for helpful comments and suggestions.

Funding

This study was funded by the National Key R&D Program of China (2019YFD1002401), the National High Technology Research and Development Program of China (863 Program) (2013AA10230402), and Agricultural Science and Technology Project of Shaanxi Province (2016NY-157).

Author information

Authors and Affiliations

  1. College of Mechanical and Electronic Engineering, Northwest A&F University, Yangling, 712100, China

    Sashuang Sun, Mei Jiang, Dongjian He, Yan Long & Huaibo Song

  2. College of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou, 310058, China

    Sashuang Sun, Ning Liang & Zhenjiang Zhou

  3. Key Laboratory of Agricultural Internet of Things, Ministry of Agriculture, Yangling, 712100, China

    Mei Jiang, Dongjian He, Yan Long & Huaibo Song

  4. Shaanxi Key Laboratory of Agricultural Information Perception and Intelligent Service, Yangling, 712100, China

    Mei Jiang, Dongjian He, Yan Long & Huaibo Song

Authors
  1. Sashuang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mei Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ning Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dongjian He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yan Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Huaibo Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhenjiang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Sashuang Sun and Huaibo Song contributed to the study conception and design. Data collection and analysis were performed by Sashuang Sun, Mei Jiang, Ning Liang, Huaibo Song and Dongjian He. Material preparation and experiment optimization were executed by Yan Long and Zhenjiang Zhou. The first draft of the manuscript was written by Sashuang Sun and Huaibo Song and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Huaibo Song.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, S., Jiang, M., Liang, N. et al. Combining an information-maximization-based attention mechanism and illumination invariance theory for the recognition of green apples in natural scenes. Multimed Tools Appl 79, 28301–28327 (2020). https://doi.org/10.1007/s11042-020-09342-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09342-2

Keywords

Search

Navigation