Using lightweight deep learning algorithm for real-time detection of apple flowers in natural environments

https://doi.org/10.1016/j.compag.2023.107765Get rights and content

Highlights

  • Apple flowers detection based on lightweight YOLOv5s was proposed.

  • YOLOv5s was used as the basic framework to detect apple flowers.

  • ShuffleNetv2 and Ghost Module were introduced to realize the lightweight of the model.

  • This method is helpful for the visual system of the flower thinning machinery.

Abstract

Flower thinning at the most appropriate stage could achieve high and stable yield of apple. Achieving the accurate and real-time detection of apple flowers can provide necessary technical support for the vision system of thinning robots. An apple flower detection method based on lightweight YOLOv5s algorithm was proposed. The original Backbone of YOLOv5s was replaced by ShuffleNetv2, and the Conv module of the Neck part of YOLOv5s network was replaced by Ghost module. ShuffleNetv2 reduced the memory access cost through Channel Split operation. Ghost module reduced the computing cost of the general volume layer while maintaining the similar detection performance. The combination of these two methods in the improvement of YOLOv5s network can greatly reduce the size of the model and improve the detection speed, which was convenient for the migration and application of the model. To verify the effectiveness of the model, 3005 apple flower images in different environments were used for training and testing. The Precision, Recall, and mean Average Precision (mAP) of YOLOv5s-ShuffleNetv2-Ghost model were 88.40 %, 86.10 %, and 91.80 %, respectively, the model size was only 0.61 MB, and the detection speed was 86.21 fps. The detection speed of YOLOv5s-ShuffleNetv2-Ghost model on the Jetson nano B01 development board was 2.48 fps. The results showed that the method was feasible for real-time and accurate detection of apple flowers. The research can provide technical reference for the development of orchard flower thinning robots.

Introduction

The density of apple flowers is closely related to fruit yield, and apple flower thinning is a key step for apple planting (Iwanami et al., 2019, Wang et al., 2020). Apple flower thinning is helpful for fruit trees to overcome the phenomenon of alternate bearing, and is also essential to improve the size, color, and taste of apples (Link, 2000, Pellerin et al., 2011). The premise of flower thinning is to realize accurate detection of apple flowers. However, there are many varieties of apples, resulting in varied shapes and colors of apple flowers. In addition, the diversities of natural environments, such as shadows, light changes, and weather changes, will also lead to the reduction of detection accuracy. Thus, realizing accurate detection of flowers in complex environments has attracted increasing attention.

Three common methods of apple flower thinning include manual thinning, chemical thinning, and mechanical thinning (Kong et al., 2009, Romano et al., 2019). Manual flower thinning requires a lot of labor. The labor cost and intensity is high, resulting in low efficiency (Farjon et al., 2020). Chemical thinning is achieved by spraying chemicals on the surface of trees, which is less expensive and more efficient, however, the spraying stage of the flower thinning reagents needs to be strictly controlled. Apple trees that were thinned at the Bud Stage, just about to bloom, produced better apples than those that were thinned four weeks after flowering. Thus, the detection of flowers in this process is extremely important (Farjon et al., 2020, Penzel et al., 2021). Mechanical thinning can significantly reduce labor intensity and production costs, and it is the main direction of the future development of thinning operation (Penzel et al., 2021). Combining image processing and machine vision technology to achieve accurate and real-time detection of apple flowers can provide necessary support for the vision system for flower thinning machinery, which is one of the cores of mechanical flower thinning technology in the future.

Traditional target detection algorithms consist of three steps: region selection, feature extraction, and classification. With the continuous development of traditional target detection algorithms, their performance tends to be saturated (Zou et al., 2019). However, traditional target detection algorithms still have the following limitations: (1) While generating candidate regions, a large number of redundant regions are also generated. (2) In complex background, it is difficult to extract representative semantic information from feature descriptors designed based on low-level visual cues (Wu et al., 2020a). Therefore, for complex natural scenes, such as complex backgrounds, severe occlusion between targets, and uneven illumination, traditional target detection algorithms can no longer meet the requirements, and traditional target detection algorithms have the disadvantages of low detection accuracy, slow detection speed, poor real-time performance, and weak universality. Deep learning-based target detection model is an end-to-end detection model that combines feature extraction, feature selection, and feature classification of targets in the same model. Deep learning models have a high hierarchical structure and strong learning ability (Kamilaris and Prenafeta-Boldú, 2018), and have unique advantages in dealing with complex visual information (Li et al., 2022).

The YOLOv5 target detection algorithm has high detection accuracy, fast inference speed, and small weight, making it suitable to be embedded on flower thinning machinery for real-time detection (Jiang et al., 2022a, Yang, 2021). The ShuffleNetv2 model proposed by Ma et al. (2018) adopted Channel Split operation. At the beginning of each unit, the input of feature channel was divided into two branches, one of which did not operate, and the other branch was composed of three Convolution modules with the same input and output channels. Channel shuffle is an operation that disrupts the channel order of the original feature map, by which the information exchange between two branches can be enhanced. The Ghost module proposed by Han et al. (2020) generated ghost feature maps through linear operation on the intrinsic feature maps, and then output both intrinsic feature maps and ghost feature maps in parallel. Compared with ordinary CNN, the total amount of parameters and computational complexity of Ghost module were reduced without changing the size of output feature maps. ShuffleNetv2 has the advantages of lightweight and fast inference speed. Ghost module can reduce the calculation cost of general Convolution layer while maintaining similar recognition performance, which is convenient for model migration.

In order to solve the problems that the model size and parameter quantity of the existing flower detection algorithm are large and the model migration is difficult, and to realize accurate and real-time detection of apple flowers, the real-time detection of apple flowers under natural environments based on lightweight YOLOv5s was proposed. The Backbone of YOLOv5s was combined with ShuffleNetv2 algorithm, while replacing the Conv of the Neck part of YOLOv5s with Ghost module to minimize the size of the model, the detection accuracy was ensured, thus achieving accurate and fast detection of apple flowers.

Up to now, there are few lightweight models for real-time detection of apple flowers. To draw on this respect, YOLOv5s-ShuffleNetv2-Ghost was proposed as a lightweight and efficient deep learning model in this study. Specific objectives were:

  • (1)

    to develop a deep learning-based network for real-time and accurate detection of apple flowers;

  • (2)

    to develop a lightweight model for real-time detection of apple flowers by combining YOLOv5s with ShuffleNetv2 and Ghost module;

  • (3)

    to test the effectiveness of the proposed model in detecting apple flowers in different scenarios and summarize its benefits and drawbacks by comparing it with other representative models.

Section snippets

Target detection based on deep learning

Traditional machine vision mainly focuses on crop classification, detection, and segmentation based on color, shape, texture, and threshold (Tian et al., 2020). In natural scenes, the variability of light, weather, and shading are the major challenges for flower detection (Farjon et al., 2020, Jiang et al., 2022b). Convolutional Neural Network (CNN)-based deep learning techniques avoid hand-designed features (Farjon et al., 2020, Triki et al., 2022), and numerous studies have shown that the

Acquisition of apple flower dataset

The subjects of this research were the flowers of more than 300 apple varieties with Fuji as the main variety. The images of apple flowers were obtained from the experimental farm attached to the College of Horticulture, Northwest A&F University, Yangling, China. The apple planting mode was dwarf close planting with row spacing of 3 m. The image acquisition device was Huawei Nova7, the image focal length was 26 mm, and the resolution was 4608 × 3456 pixels. The shooting period was from March

Results and analysis

In order to evaluate the performance of the apple flower detection model proposed in this study, the network was trained using the training set. In addition, on the test set, the YOLOv5s-ShuffleNetv2-Ghost model proposed in this study was compared with other models including Faster R-CNN (Ren et al., 2017), SSD (Liu et al., 2016), YOLOv4 (Bochkovskiy et al., 2020), YOLOv5s, YOLOv5s-MobileNetv2-Ghost, and YOLOv7 (Wang et al., 2022).

Impact of different modules on improved algorithms

In order to explore the influence of ShuffleNetv2 and Ghost module on the model, the ablation experiment was designed in this study. ShuffleNetv2 and Ghost module were removed respectively to obtain YOLOv5s-Ghost model and YOLOv5s-ShuffleNetv2 model. These two models were used to train the training set of apple flowers, and the number of network training epochs was 300, then the test set was used to evaluate the performance of the two detection algorithms. Table 3 lists the performance

Conclusion

For the accurate and real-time detection of apple flowers in field, a method based on lightweight YOLOv5s model was proposed in this study. The main conclusions of this study can be summarized as follows:

  • (1)

    The YOLOv5s-ShuffleNetv2-Ghost algorithm proposed in this study replaced the Backbone of YOLOv5s with ShuffleNetv2, and replaced the C3 and Conv modules in the Neck of YOLOV5s with Ghost module. The algorithm had obvious advantages in terms of model size and detection speed, which can greatly

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by the National Key R&D Program of China (2019YFD1002401), and the National Natural Science Foundation of China (31701326). The authors would like to thank all of the authors cited in this article and the anonymous referees for their helpful comments and suggestions.

References (44)

  • Y. Tian et al.

    Instance segmentation of apple flowers using the improved Mask R-CNN model

    Biosyst. Eng.

    (2020)
  • A. Triki et al.

    A deep learning-based approach for detecting plant organs from digitized herbarium specimen images

    Ecol. Inform.

    (2022)
  • D. Wang et al.

    Channel pruned YOLOV5s-based deep learning approach for rapid and accurate apple fruitlet detection before fruit thinning

    Biosyst. Eng.

    (2021)
  • X.A. Wang et al.

    Side-view apple flower mapping using edge-based fully convolutional networks for variable rate chemical thinning

    Comput. Electron. Agric.

    (2020)
  • X.A. Wang et al.

    DeepPhenology: Estimation of apple flower phenology distributions based on deep learning

    Comput. Electron. Agric.

    (2021)
  • D. Wu et al.

    Using channel pruning-based YOLOv4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments

    Comput. Electron. Agric.

    (2020)
  • X. Wu et al.

    Recent advances in deep learning for object detection

    Neurocomputing

    (2020)
  • Bochkovskiy, A., Wang, C.Y., Liao, H. Y. M., 2020. YOLOv4: Optimal speed and accuracy of object detection. arXiv...
  • G. Farjon et al.

    Detection and counting of flowers on apple trees for better chemical thinning decisions

    Precis. Agric.

    (2020)
  • B. Fpa et al.

    Automated grapevine flower detection and quantification method based on computer vision and deep learning from on-the-go imaging using a mobile sensing platform under field conditions

    Comput. Electron. Agric.

    (2020)
  • L.A. Guo et al.

    Real-time detection of kiwifruit flower and bud simultaneously in orchard using YOLOv4 for robotic pollination

    Comput. Electron. Agric.

    (2022)
  • K. Han et al.

    Ghostnet: More features from cheap operations

  • Cited by (21)

    View all citing articles on Scopus
    View full text