Real-time strawberry detection using deep neural networks on embedded system (rtsd-net): An edge AI application
Introduction
Machine learning is a hot topic in the field of artificial intelligence and it is the key technique for smart agricultural machinery. Deep learning is the forefront of the development of machine learning which has achieved excellent results in many ways like natural language processing and computer vision. Deep convolutional neural networks (DCNNs) have made significant progress in computer vision, which have greatly improved the performance of image classification, segmentation and detection. In terms of object detection, the accuracy of deep learning algorithms can be higher than visual recognition with incomparable efficiency. In the latest results, Deep learning algorithms have achieved 97% accuracy in the identification and positioning of rhizome features (Pound et al., 2018).
The current development of deep learning has two directions. One is to develop towards high precision, and the other is to develop towards high efficiency. Alexey AB (Bochkovskiy et al., 2020) proposed YOLOv4 with more prominent accuracy on the basis of YOLOv3, which improved data processing, network structure and loss function to achieve better detection accuracy. In addition to the accuracy improvement, the improvement in speed is also of great significance. Andrew (Howard et al., 2017) proposed the deep separable convolution network, and constructed the lightweight network MobileNet-v2 to greatly reduce the network parameters. In terms of the results, the accuracy of the network has declined to a certain extent while the detection speed has been greatly improved, which made it meet the requirements of specific scenarios. Based on YOLOv4 network, Alexey A.B proposed YOLOv4-tiny lightweight network, whose speed can reach 371 FPS when running on PC with GTX 1080 Ti. Besides, the model can also be deployed on embedded mobile devices, such as Jetson Nano or mobile smart phone. Lightweight network greatly promotes the deployment process of the model on edge computing devices.
Computer vision has experienced great achievement recently in agriculture automation and deep learning. For example, Chen (M. (Chen et al., 2021) obtained a detailed 3D orchard map combining eye-hand stereo vision and SLAM system. The candidate fruits are obtained by partial matching of contour information and probabilistic Hough transform, and then trained by SVM classifier. The experiment has good detection effect in most types of fruits (Lin et al., 2019). However, in smart agricultural machinery, many tasks need to be executed quickly on mobile devices like embedded systems, which makes researchers pay attention to the development of light weight deep neural network in edge computing. Deep learning algorithm has been used to identify crops, weeds and diseases in agriculture. (Y. (Chen et al., 2019) used UAV and RGB cameras to shoot strawberry fields, and used Faster-RCNN algorithm to detect the targets after segmentation. The number of strawberries and flowers was calculated to estimate the yield and accuracy rate has reached 84.1%. This research helps farmers to observe the growth cycle of strawberry fields in a more efficient way. (Gao et al., 2020) conducted research based on YOLOv3-tiny network to improve the speed of object detection and manage weeds in sugar beet field, which meets the requirements of speed and accuracy. Liu (Liu and Wang, 2020) proposes an early recognition method of tomato leaf spot based on MobileNetv2-YOLOv3 model to achieve a good balance between the accuracy and real-time detection of tomato gray leaf spot. Aguiar used deep learning algorithms such as MobileNets, Inception and lite YOLO to detect the trunk in real-time detection as an experimental performance benchmark between two devices is performed: NVIDIA’s Jetson Nano and Google’s USB Accelerator (Aguiar et al., 2020). Mazzia carried out the benchmark tests on two experimental devices, the Jetson Nano of NVIDIA and the USB Accelerator of Google for the real-time detection of apples in orchards and the evaluation of apple production (Mazzia et al., 2020).
Strawberries are of good taste and rich nutritional value, like multiple vitamins and amino acids(Proteggente et al., 2002). It’s well loved by the people all over the world. The worldwide consumption is expected to reach 11.5million tons. Among all producing countries China has the largest production and consumption. However, this industry is very labor-intensive, especially in harvesting, which makes the price of strawberries so high. Strawberry fruits should be harvested as soon as possible after maturity, but the problem is, the workload is so heavy that it is impossible to finish the harvest in such a short time. So, it is very meaningful to use automatic equipment to harvest strawberry. From this aspect, reducing labor force can really bring significant economic benefits
One key technique for smart harvesting is real-time object detection. How to improve the recognition speed of strawberries while ensuring the recognition accuracy of strawberries has become the focus of this work. Real-time detection algorithm is of great research significance in agriculture. In the single agricultural scene, the variety of object is far lower than some public datasets, such as COCO datasets. In the case of simple scene and fewer types of objects, how does reducing the complexity of network structure affect the accuracy and speed of object detection? In intelligent agricultural machinery, the improvement of detection speed is valuable. Although some researchers work on edge computing methods to deploy deep neural network on mobile platform, few of them works on the CNN structure to improve performance. The deep neural network consists of millions of parameters and sophisticated neural structures. Which parameter can be reduced and how is the effect on efficiency? This is a big issue in building DNN since some change may lead to bad effect.
In this research, we aim to find out a way to improve the performance of lightweight deep neural network and deploy it on embedded platform. And we aim to find out what can be done to the internal structure of YOLOv4-tiny so the speed can increase without compromising accuracy.
Section snippets
System setup
As the Fig. 1 shows, UAV was used for fast collection of data over the strawberry field. A total of 3 categories, strawberry, immature strawberry and flower were marked. Before the data set was put into the network, the data set needs preprocessing. On the network structure, we compared the detection results of 8 kinds of networks. 4 of them are YOLOv3, YOLOv3-tiny, YOLOv4, and YOLOv4-tiny, and the rest are the improved network structures. The corresponding speed was obtained by inference on PC
Accuracy
The YOLO algorithm shows good performance on the COCO data set. The mAP and F1 values of YOLOv3 and YOLOv4 are higher than the YOLOv3-tiny and YOLOv4-tiny because YOLOv3 and YOLOv4 have more anchors to detect targets. We tested the mAP and F1 of 4 different YOLO models on the strawberry data set, as shown in Table 5. YOLOv3-tiny and YOLOv4-tiny are lightweight deep learning networks. The mAP of the YOLOv3-tiny model is significantly lower than the other 3 models. The mAP of YOLOv4-tiny is
Conclusion
Among the four existing YOLO networks, YOLOv4-tiny is the state-of-art deep neural network in object detection for edge computing as it is basically the same as YOLOv3 and YOLOv4 but it’s much faster than the two model. So, it’s selected as the basis of this research for further modifications to meet the demand of real-time strawberry detection. By gradually reducing the complexity of the network, the network parameters are reduced and the detection speed is improved. Among the 4 modified
Funding
This research was funded by the National Natural Science Foundation of China (NSFC), grant number: 61905219.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (25)
- et al.
3D global mapping of large-scale unstructured orchard integrating eye-in-hand stereo vision and SLAM
Comput. Electron. Agric.
(2021) - et al.
Vineyard trunk detection using deep learning – An experimental device benchmark
Comput. Electron. Agric.
(2020) - Bochkovskiy, A., Wang, C.-Y., & Liao, H.-Y. M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv...
Nvidia makes it easy to embed AI: The Jetson nano packs a lot of machine-learning power into DIY projects-[Hands on]
IEEE Spectr.
(2020)- et al.
Strawberry Yield Prediction Based on a Deep Neural Network Using High-Resolution Aerial Orthoimages
Remote Sens.
(2019) - DeVries, T., & Taylor, G.W., 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint...
- et al.
Deep convolutional neural networks for image-based Convolvulus sepium detection in sugar beet fields
Plant Methods
(2020) - et al.
Spatial pyramid pooling in deep convolutional networks for visual recognition
IEEE Trans. Pattern Anal. Mach. Intell.
(2015) - et al.
- Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H., 2017. MobileNets:...
Fruit detection in natural environment using partial shape matching and probabilistic Hough transform
Precis. Agric.
Early recognition of tomato gray leaf spot disease based on MobileNetv2-YOLOv3 model
Plant Methods
Cited by (73)
CottonSense: A high-throughput field phenotyping system for cotton fruit segmentation and enumeration on edge devices
2024, Computers and Electronics in AgricultureDefect detection of the surface of wind turbine blades combining attention mechanism
2024, Advanced Engineering InformaticsAn improved YOLO algorithm for detecting flowers and fruits on strawberry seedlings
2024, Biosystems EngineeringAn efficient deep learning model using network pruning for fake banknote recognition
2023, Expert Systems with ApplicationsRemote safety system for a robot tractor using a monocular camera and a YOLO-based method
2023, Computers and Electronics in AgricultureFruit sizing using AI: A review of methods and challenges
2023, Postharvest Biology and Technology