Skip to main content
SpringerLink
Log in

Kernel pooling feature representation of pre-trained convolutional neural networks for leaf recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Due to the presence of various types of factors, such as illumination, viewpoint, intra-class complexity, and inter-class similarity, which make plant leaf recognition still a challenging research problem. In this paper, we present a very simple, yet effective feature representation method for plant leaf recognition. Concretely, it comprises four stages. Firstly, each leaf image is fed into an imagenet pre-trained CNN model to extract activated feature maps in a specified layer. Secondly, inspired by 1\(\times\)1 convolution, we exploit principle component analysis to learn the 1\(\times\)1 convolution filters. As a result, it not only eliminates the redundant information, reduces the feature dimension adaptively that is beneficial to the subsequent high order pooling, but also increases classification accuracy. Thirdly, kernel pooling is employed to capture second order statistics between each pair of features with the purpose of learning more discriminative information. Finally, matrix sqrt and upper triangle are performed to obtain the final leaf representation, which is utilized for classification and retrieval by the euclidean distance based nearest neighbor classifier. Extensive experiments are conducted on four representative plant leaf datasets, Flavia, Swedish, MEW2012, ICL, to validate the effectiveness of our method. For classification task, our method achieves outstanding and better average classification accuracies than the comparative state-of-the-art baselines. For retrieval task, our method gets significant higher or competitive MAP scores. Our implementation code will be available at https://github.com/fengshu666666/leafrecognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. download link: http://flavia.sourceforge.net

  2. download link: https://www.cvl.isy.liu.se/en/research/datasets/swedish-leaf/

  3. download link: http://zoi.utia.cas.cz/node/662

  4. download link: https://download.csdn.net/download/qq_21175275/12011744

References

  1. Adamek T, O’Connor NE (2004) A multiscale representation method for nonrigid shapes with a single closed contour. IEEE Transactions on Circuits and Systems for Video Technology 14(5):742–753

    Article  Google Scholar 

  2. Alajlan N, El Rube I, Kamel MS, Freeman G (2007) Shape retrieval using triangle-area representation and dynamic space warping. Pattern Recognition 40(7):1911–1920

    Article  MATH  Google Scholar 

  3. Araujo VM, Jr, ASB, Oliveira LES, Koerich AL (2020) Two-view fine-grained classification of plant species

  4. Arribas JI, Sánchez-Ferrero GV, Ruiz-Ruiz G, Gómez-Gil J (2011) Leaf classification in sunflower crops by computer vision and neural networks. Computers and Electronics in Agriculture 78(1):9–18

    Article  Google Scholar 

  5. Bai X, Wang B, Yao C, Liu W, Tu Z (2012) Co-transduction for shape retrieval. IEEE Transactions on Image Processing 21(5):2747–2757

    Article  MathSciNet  MATH  Google Scholar 

  6. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(4):509–522

    Article  Google Scholar 

  7. Carreira J, Caseiro R, Batista J, Sminchisescu C (2015) Free-form region description with second-order pooling. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(6):1177–1189

    Article  Google Scholar 

  8. Chatfield K, Simonyan K, Vedaldi A, Zisserman A (2014) Return of the devil in the details: Delving deep into convolutional nets. In: Proceedings of the british machine vision conference (BMVC), pp 1–12

  9. Chen X, Wang B (2020) Invariant leaf image recognition with histogram of gaussian convolution vectors. Computers and Electronics in Agriculture 178:105714

    Article  Google Scholar 

  10. Dyrmann M, Karstoft H, Midtiby HS (2016) Plant species classification using deep convolutional neural network. Biosystems Engineering 151:72–80

    Article  Google Scholar 

  11. Engin M, Wang L, Zhou L, Liu X (2018) Deepkspd: Learning kernel-matrix-based spd representation for fine-grained image recognition. In: European conference on computer vision (ECCV), pp 629–645

  12. Gao Y, Beijbom O, Zhang N, Darrell T (2016) Compact bilinear pooling. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 317–326

  13. Gao Z, Xie J, Wang Q, Li P (2019) Global second-order pooling convolutional networks. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 3019–3028

  14. Grinblat GL, Uzal LC, Larese MG, Granitto PM (2016) Deep learning for plant identification using vein morphological patterns. Computers and Electronics in Agriculture 127:418–424

    Article  Google Scholar 

  15. Gupta A, Prakash D (2020) A fast and efficient color model for automatic monitoring of plants based on leaf images. Journal of Critical Reviews 7(17):2398–2404

    Google Scholar 

  16. Haykin S (1998) Neural networks: A comprehensive foundation. Prentice Hall PTR, USA, 2nd edn

  17. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 770–778

  18. He N, Fang L, Li S, Plaza A, Plaza J (2018) Remote sensing scene classification using multilayer stacked covariance pooling. IEEE Transactions on Geoscience and Remote Sensing 56(12):6899–6910

    Article  Google Scholar 

  19. Horaisová K, Kukal J (2016) Leaf classification from binary image via artificial intelligence. Biosystems Engineering 142:83–100

    Article  Google Scholar 

  20. Hu J, Chen Z, Yang M, Zhang R, Cui Y (2018) A multiscale fusion convolutional neural network for plant leaf recognition. IEEE Signal Processing Letters 25(6):853–857

    Article  Google Scholar 

  21. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(8):2011–2023

    Article  Google Scholar 

  22. Hu R, Jia W, Ling H, Huang D (2012) Multiscale distance matrix for fast plant leaf recognition. IEEE Transactions on Image Processing 21(11):4667–4672

    Article  MathSciNet  MATH  Google Scholar 

  23. Ionescu C, Vantzos O, Sminchisescu C (2015) Matrix backpropagation for deep networks with structured layers. In: IEEE International conference on computer vision (ICCV), pp 2965–2973

  24. Kaya A, Keceli AS, Catal C, Yalic HY, Temucin H, Tekinerdogan B (2019) Analysis of transfer learning for deep neural network based plant classification models. Computers and Electronics in Agriculture 158:20–29

    Article  Google Scholar 

  25. Koniusz P, Tas Y, Porikli F (2017) Domain adaptation by mixture of alignments of second-or higher-order scatter tensors. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 7139–7148

  26. Koniusz P, Yan F, Gosselin P, Mikolajczyk K (2017) Higher-order occurrence pooling for bags-of-words: Visual concept detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(2):313–326

    Article  Google Scholar 

  27. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutionalneural networks. Advances in neural information processing systems 25:1097–1105

    Google Scholar 

  28. Laga H, Kurtek S, Srivastava A, Golzarian M, Miklavcic SJ (2012) A riemannian elastic metric for shape-based plant leaf classification. In: International conference on digital image computing techniques and applications (DICTA), pp 1–7

  29. Latecki LJ, Lakamper R, Eckhardt T (2000) Shape descriptors for non-rigid shapes with a single closed contour. Proceedings IEEE conference on computer vision and pattern recognition (CVPR) 1:424–429

    Google Scholar 

  30. Lee SH, Chan CS, Remagnino P (2018) Multi-organ plant classification based on convolutional and recurrent neural networks. IEEE Transactions on Image Processing 27(9):4287–4301

    Article  MathSciNet  Google Scholar 

  31. Lee SH, Chan CS, Wilkin P, Remagnino P (2015) Deep-plant: Plant identification with convolutional neural networks. In: IEEE International conference on image processing (ICIP), pp 452–456

  32. Lee SH, Chan CS, Mayo SJ, Remagnino P (2017) How deep learning extracts and learns leaf features for plant classification. Pattern Recognition 71:1–13

    Article  Google Scholar 

  33. Li P, Xie J, Wang Q, Gao Z (2018) Towards faster training of global covariance pooling networks by iterativematrix square root normalization. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 947–955

  34. Lin T, RoyChowdhury A, Maji S (2018) Bilinear convolutional neural networks for fine-grained visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(6):1309–1322

    Article  Google Scholar 

  35. Lin TY, Maji S (2017) Improved bilinear pooling with cnns. In: British machinevision conference (BMVC), pp 1–12

  36. Ling H, Jacobs DW (2007) Shape classification using the inner-distance. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(2):286–299

    Article  Google Scholar 

  37. Liu N, Kan J (2016) Improved deep belief networks and multi-feature fusion for leaf identification. Neurocomputing 216:460–467

    Article  Google Scholar 

  38. Mehdipour Ghazi M, Yanikoglu B, Aptoula E (2017) Plant identification using deep neural networks via optimization of transfer learning parameters. Neurocomputing 235:228–235

    Article  Google Scholar 

  39. Novotny P, Suk T (2013) Leaf recognition of woody species in central europe. Biosystems Engineering 115(4):444–452

    Article  Google Scholar 

  40. Raina S, Gupta A (2021) A study on various techniques for plant leaf disease detection using leaf image. In: International conference on artificial intelligence and smart systems (ICAIS), pp 900–905

  41. Sachar S, Kumar A (2021) Survey of feature extraction and classification techniques to identify plant through leaves. Expert Systems with Applications 167:114181

    Article  Google Scholar 

  42. Shah MP, Singha S, Awate SP (2017) Leaf classification using marginalized shape context and shape+texture dual-path deep convolutional neural network. In: IEEE International conference on image processing (ICIP), pp 860–864

  43. Shao Y (2019) Supervised global-locality preserving projection for plant leaf recognition. Computers and Electronics in Agriculture 158:102–108

    Article  Google Scholar 

  44. Sharma S, Gupta A (2020) A review for the automatic methods of plant’s leaf image segmentation. International Journal of Intelligence and Sustainable Computing 1(1):101–114

    Article  Google Scholar 

  45. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (ICLR), pp 1–14

  46. Soderkvist OJO (2001) Computer vision classifcation of leaves from swedish trees. Master’s thesis, Linkoping University

  47. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 1–9

  48. Trivedi M, Gupta A (2021) Automatic monitoring of the growth of plants using deep learning-based leaf segmentation. International Journal of Applied Science and Engineering 18(2):1–9

    Google Scholar 

  49. Turkoglu M, Hanbay D (2019) Recognition of plant leaves: An approach with hybrid features produced by dividing leaf images into two and four parts. Applied Mathematics and Computation 352:1–14

    Article  MathSciNet  MATH  Google Scholar 

  50. Tuzel O, Porikli F, Meer P (2006) Region covariance: A fast descriptor for detection and classification. European conference on computer vision (ECCV). Springer, Berlin, pp 589–600

  51. Vedaldi A, Lenc K (2015) Matconvnet convolutional neural networks for matlab. In: Proceedings of the 23rd ACM international conference on multimedia, pp 689–692

  52. Wang B, Gao Y (2014) Hierarchical string cuts: A translation, rotation, scale, and mirror invariant descriptor for fast shape retrieval. IEEE Transactions on Image Processing 23(9):4101–4111

    Article  MathSciNet  MATH  Google Scholar 

  53. Wang B, Gao Y, Sun C, Blumenstein M, La Salle J (2017) Can walking and measuring along chord bunches better describe leaf shapes? In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 2047–2056

  54. Wang B, Wang D (2019) Plant leaves classification: A few-shot learning method based on siamese network. IEEE Access 7:151754–151763

    Article  Google Scholar 

  55. Wang B, Brown D, Gao Y, Salle JL (2015) March: Multiscale-arch-height description for mobile retrieval of leaf images. Information Sciences 302:132–148

    Article  Google Scholar 

  56. Wang J, Bai X, You X, Liu W, Latecki LJ (2012) Shape matching and classification using height functions. Pattern Recognition Letters 33(2):134–143

    Article  Google Scholar 

  57. Wang X, Du W, Guo F, Hu S (2020) Leaf recognition based on elliptical half gabor and maximum gap local line direction pattern. IEEE Access 8:39175–39183

    Article  Google Scholar 

  58. Wang X, Feng B, Bai X, Liu W, Jan Latecki L (2014) Bag of contour fragments for robust shape classification. Pattern Recognition 47(6):2116–2125

    Article  Google Scholar 

  59. Wang Z, Sun X, Yang Z, Zhang Y, Zhu Y, Ma Y (2018) Leaf recognition based on dpcnn and bow. Neural Processing Letters 47(1):99–115

    Article  Google Scholar 

  60. Wu SG, Bao FS, Xu EY, Wang Y, Chang Y, Xiang Q (2007) A leaf recognition algorithm for plant classification using probabilistic neural network. In: IEEE International symposium on signal processing and information technology, pp 11–16

  61. Yang C (2021) Plant leaf recognition by integrating shape and texture features. Pattern Recognition 112:107809

    Article  Google Scholar 

  62. Yang C, Wei H, Yu Q (2016) Multiscale triangular centroid distance for shape-based plant leaf recognition. In: Proceedings of the twenty-second european conference on artificial intelligence (ECAI), pp 269–276

  63. Yang C, Yu Q (2019) Multiscale fourier descriptor based on triangular features for shape retrieval. Signal Processing: Image Communication 71:110–119

    Google Scholar 

  64. Yousefi E, Baleghi Y, Sakhaei SM (2017) Rotation invariant wavelet descriptors, a new set of features to enhance plant leaves classification. Computers and Electronics in Agriculture 140:70–76

    Article  Google Scholar 

  65. Zeng S, Zhang B, Du Y (2017) Joint distances by sparse representation and locality-constrained dictionary learning for robust leaf recognition. Computers and Electronics in Agriculture 142:563–571

    Article  Google Scholar 

  66. Zhang J, Wang L, Zhou L, Li W (2021) Beyond covariance: Sice and kernel based visual feature representation. International Journal of Computer Vision 129(2):300–320

    Article  Google Scholar 

  67. Zhang S, Huang W, an Huang Y, Zhang C (2020) Plant species recognition methods using leaf image: Overview. Neurocomputing 408:246–272

    Article  Google Scholar 

  68. Zhang S, Lei Y, Zhang C, Hu Y (2016) Semi-supervised orthogonal discriminant projection for plant leaf classification. Pattern Analysis and Applications 19:953–961

    Article  MathSciNet  Google Scholar 

  69. Zhang S, Zhang C, Wang X (2020) Plant species recognition based on global-local maximum margin discriminant projection. Knowledge-Based Systems 200:105998

    Article  Google Scholar 

  70. Zhang X, Zhao W, Luo H, Chen L, Peng J, Fan J (2019) Plant recognition via leaf shape and margin features. Multimedia Tools and Applications 78(19):27463–27489

    Article  Google Scholar 

  71. Zhao C, Chan SS, Cham WK, Chu L (2015) Plant identification using leaf shapes - a pattern counting approach. Pattern Recognition 48(10):3203–3215

    Article  Google Scholar 

  72. Zhou J, Qi H, Chen Y, Wang H (2021) Progressive principle component analysis for compressing deep convolutional neural networks. Neurocomputing 440:197–206

    Article  Google Scholar 

Download references

Funding

The work described in this paper was partially supported by the National Natural Science Foundation of China (Grant No. 62101376), Science and Technology Innovation Foundation Project of Shanxi Agricultural University (Grant Number 2018021), Natural Science Foundation of Shanxi Province of China (Grant No. 201901D211078), Scientific and Technologial Innovation Programs of Higher Education Institutions in Shanxi (Grant No. 2019L0350).

Author information

Authors and Affiliations

  1. Department of Foundation, Shanxi Agricultural University, 030801, Shanxi, China

    Shu Feng

Authors
  1. Shu Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shu Feng.

Ethics declarations

Conflicts of interest

We declare that there have no conflicts of interest to this work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, S. Kernel pooling feature representation of pre-trained convolutional neural networks for leaf recognition. Multimed Tools Appl 81, 4255–4282 (2022). https://doi.org/10.1007/s11042-021-11769-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11769-0

Keywords

Search

Navigation