Using lightweight deep learning algorithm for real-time detection of apple flowers in natural environments
Introduction
The density of apple flowers is closely related to fruit yield, and apple flower thinning is a key step for apple planting (Iwanami et al., 2019, Wang et al., 2020). Apple flower thinning is helpful for fruit trees to overcome the phenomenon of alternate bearing, and is also essential to improve the size, color, and taste of apples (Link, 2000, Pellerin et al., 2011). The premise of flower thinning is to realize accurate detection of apple flowers. However, there are many varieties of apples, resulting in varied shapes and colors of apple flowers. In addition, the diversities of natural environments, such as shadows, light changes, and weather changes, will also lead to the reduction of detection accuracy. Thus, realizing accurate detection of flowers in complex environments has attracted increasing attention.
Three common methods of apple flower thinning include manual thinning, chemical thinning, and mechanical thinning (Kong et al., 2009, Romano et al., 2019). Manual flower thinning requires a lot of labor. The labor cost and intensity is high, resulting in low efficiency (Farjon et al., 2020). Chemical thinning is achieved by spraying chemicals on the surface of trees, which is less expensive and more efficient, however, the spraying stage of the flower thinning reagents needs to be strictly controlled. Apple trees that were thinned at the Bud Stage, just about to bloom, produced better apples than those that were thinned four weeks after flowering. Thus, the detection of flowers in this process is extremely important (Farjon et al., 2020, Penzel et al., 2021). Mechanical thinning can significantly reduce labor intensity and production costs, and it is the main direction of the future development of thinning operation (Penzel et al., 2021). Combining image processing and machine vision technology to achieve accurate and real-time detection of apple flowers can provide necessary support for the vision system for flower thinning machinery, which is one of the cores of mechanical flower thinning technology in the future.
Traditional target detection algorithms consist of three steps: region selection, feature extraction, and classification. With the continuous development of traditional target detection algorithms, their performance tends to be saturated (Zou et al., 2019). However, traditional target detection algorithms still have the following limitations: (1) While generating candidate regions, a large number of redundant regions are also generated. (2) In complex background, it is difficult to extract representative semantic information from feature descriptors designed based on low-level visual cues (Wu et al., 2020a). Therefore, for complex natural scenes, such as complex backgrounds, severe occlusion between targets, and uneven illumination, traditional target detection algorithms can no longer meet the requirements, and traditional target detection algorithms have the disadvantages of low detection accuracy, slow detection speed, poor real-time performance, and weak universality. Deep learning-based target detection model is an end-to-end detection model that combines feature extraction, feature selection, and feature classification of targets in the same model. Deep learning models have a high hierarchical structure and strong learning ability (Kamilaris and Prenafeta-Boldú, 2018), and have unique advantages in dealing with complex visual information (Li et al., 2022).
The YOLOv5 target detection algorithm has high detection accuracy, fast inference speed, and small weight, making it suitable to be embedded on flower thinning machinery for real-time detection (Jiang et al., 2022a, Yang, 2021). The ShuffleNetv2 model proposed by Ma et al. (2018) adopted Channel Split operation. At the beginning of each unit, the input of feature channel was divided into two branches, one of which did not operate, and the other branch was composed of three Convolution modules with the same input and output channels. Channel shuffle is an operation that disrupts the channel order of the original feature map, by which the information exchange between two branches can be enhanced. The Ghost module proposed by Han et al. (2020) generated ghost feature maps through linear operation on the intrinsic feature maps, and then output both intrinsic feature maps and ghost feature maps in parallel. Compared with ordinary CNN, the total amount of parameters and computational complexity of Ghost module were reduced without changing the size of output feature maps. ShuffleNetv2 has the advantages of lightweight and fast inference speed. Ghost module can reduce the calculation cost of general Convolution layer while maintaining similar recognition performance, which is convenient for model migration.
In order to solve the problems that the model size and parameter quantity of the existing flower detection algorithm are large and the model migration is difficult, and to realize accurate and real-time detection of apple flowers, the real-time detection of apple flowers under natural environments based on lightweight YOLOv5s was proposed. The Backbone of YOLOv5s was combined with ShuffleNetv2 algorithm, while replacing the Conv of the Neck part of YOLOv5s with Ghost module to minimize the size of the model, the detection accuracy was ensured, thus achieving accurate and fast detection of apple flowers.
Up to now, there are few lightweight models for real-time detection of apple flowers. To draw on this respect, YOLOv5s-ShuffleNetv2-Ghost was proposed as a lightweight and efficient deep learning model in this study. Specific objectives were:
- (1)
to develop a deep learning-based network for real-time and accurate detection of apple flowers;
- (2)
to develop a lightweight model for real-time detection of apple flowers by combining YOLOv5s with ShuffleNetv2 and Ghost module;
- (3)
to test the effectiveness of the proposed model in detecting apple flowers in different scenarios and summarize its benefits and drawbacks by comparing it with other representative models.
Section snippets
Target detection based on deep learning
Traditional machine vision mainly focuses on crop classification, detection, and segmentation based on color, shape, texture, and threshold (Tian et al., 2020). In natural scenes, the variability of light, weather, and shading are the major challenges for flower detection (Farjon et al., 2020, Jiang et al., 2022b). Convolutional Neural Network (CNN)-based deep learning techniques avoid hand-designed features (Farjon et al., 2020, Triki et al., 2022), and numerous studies have shown that the
Acquisition of apple flower dataset
The subjects of this research were the flowers of more than 300 apple varieties with Fuji as the main variety. The images of apple flowers were obtained from the experimental farm attached to the College of Horticulture, Northwest A&F University, Yangling, China. The apple planting mode was dwarf close planting with row spacing of 3 m. The image acquisition device was Huawei Nova7, the image focal length was 26 mm, and the resolution was 4608 × 3456 pixels. The shooting period was from March
Results and analysis
In order to evaluate the performance of the apple flower detection model proposed in this study, the network was trained using the training set. In addition, on the test set, the YOLOv5s-ShuffleNetv2-Ghost model proposed in this study was compared with other models including Faster R-CNN (Ren et al., 2017), SSD (Liu et al., 2016), YOLOv4 (Bochkovskiy et al., 2020), YOLOv5s, YOLOv5s-MobileNetv2-Ghost, and YOLOv7 (Wang et al., 2022).
Impact of different modules on improved algorithms
In order to explore the influence of ShuffleNetv2 and Ghost module on the model, the ablation experiment was designed in this study. ShuffleNetv2 and Ghost module were removed respectively to obtain YOLOv5s-Ghost model and YOLOv5s-ShuffleNetv2 model. These two models were used to train the training set of apple flowers, and the number of network training epochs was 300, then the test set was used to evaluate the performance of the two detection algorithms. Table 3 lists the performance
Conclusion
For the accurate and real-time detection of apple flowers in field, a method based on lightweight YOLOv5s model was proposed in this study. The main conclusions of this study can be summarized as follows:
- (1)
The YOLOv5s-ShuffleNetv2-Ghost algorithm proposed in this study replaced the Backbone of YOLOv5s with ShuffleNetv2, and replaced the C3 and Conv modules in the Neck of YOLOV5s with Ghost module. The algorithm had obvious advantages in terms of model size and detection speed, which can greatly
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported by the National Key R&D Program of China (2019YFD1002401), and the National Natural Science Foundation of China (31701326). The authors would like to thank all of the authors cited in this article and the anonymous referees for their helpful comments and suggestions.
References (44)
- et al.
Real-time, highly accurate robotic grasp detection utilizing transfer learning for robots manipulating fragile fruits with widely variable sizes and shapes
Comput. Electron. Agric.
(2022) - et al.
Apple flower detection using deep convolutional networks
Comput. Ind.
(2018) - et al.
Real-time defects detection for apple sorting using NIR cameras with pruning-based YOLOv4 network
Comput. Electron. Agric.
(2022) - et al.
Apple thinning strategy based on a model predicting flower-bud formation
Sci. Hortic.
(2019) - et al.
A review of YOLO algorithm developments
Procedia. Comput. Sci.
(2022) - et al.
Deep neural network based date palm tree detection in drone imagery
Comput. Electron. Agric.
(2022) - et al.
Deep learning in agriculture: A survey
Comput. Electron. Agric.
(2018) - et al.
Fast detection and location of longan fruits using UAV images
Comput. Electron. Agric.
(2021) - et al.
Fruit detection and load estimation of an orange orchard using the YOLO models through simple approaches in different imaging and illumination conditions
Comput. Electron. Agric.
(2021) - et al.
Apple, peach, and pear flower detection using semantic segmentation network and shape constraint level set
Comput. Electron. Agric.
(2021)
Instance segmentation of apple flowers using the improved Mask R-CNN model
Biosyst. Eng.
A deep learning-based approach for detecting plant organs from digitized herbarium specimen images
Ecol. Inform.
Channel pruned YOLOV5s-based deep learning approach for rapid and accurate apple fruitlet detection before fruit thinning
Biosyst. Eng.
Side-view apple flower mapping using edge-based fully convolutional networks for variable rate chemical thinning
Comput. Electron. Agric.
DeepPhenology: Estimation of apple flower phenology distributions based on deep learning
Comput. Electron. Agric.
Using channel pruning-based YOLOv4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments
Comput. Electron. Agric.
Recent advances in deep learning for object detection
Neurocomputing
Detection and counting of flowers on apple trees for better chemical thinning decisions
Precis. Agric.
Automated grapevine flower detection and quantification method based on computer vision and deep learning from on-the-go imaging using a mobile sensing platform under field conditions
Comput. Electron. Agric.
Real-time detection of kiwifruit flower and bud simultaneously in orchard using YOLOv4 for robotic pollination
Comput. Electron. Agric.
Ghostnet: More features from cheap operations
Cited by (21)
Automated measurement of beef cattle body size via key point detection and monocular depth estimation
2024, Expert Systems with ApplicationsMonitoring of key Camellia Oleifera phenology features using field cameras and deep learning
2024, Computers and Electronics in AgricultureUsing an improved lightweight YOLOv8 model for real-time detection of multi-stage apple fruit in complex orchard environments
2024, Artificial Intelligence in AgricultureAD-YOLOv5: An object detection approach for key parts of sika deer based on deep learning
2024, Computers and Electronics in AgricultureA fast and data-efficient deep learning framework for multi-class fruit blossom detection
2024, Computers and Electronics in AgricultureA fast and lightweight detection model for wheat fusarium head blight spikes in natural environments
2024, Computers and Electronics in Agriculture