Real-time strawberry detection using deep neural networks on embedded system (rtsd-net): An edge AI application

https://doi.org/10.1016/j.compag.2021.106586Get rights and content

Highlights

Abstract

Computer vision is a key technique to make agricultural machinery smart. Deep neural network has achieved great success in computer vision. How to use it at a small size, low cost, low power consumption device with high accuracy and speed on strawberry harvesting machinery has drawn much research attention. Since the infield situation has reduced number of objects and that they are easier to be distinguished from the background compared to other computer vision datasets, the huge neural network structure can be simplified in order to speed up the detection inference without penalizing the detection accuracy. In this research, a new deep neural network called RTSD-Net is proposed based on stat-of-art light-weighted YOLOv4-tiny with reduced layers and modified structure for real-time strawberry detection under infield condition. The original CSPNet was replaced by 2 types of CSPNet designed with reduced parameters and a simplified structure and 4 new network structures are designed by combining these 2 types. The performances of the 4 networks were evaluated. It was observed that the number of parameters of these 4 networks and the detection speed of the model is negatively correlated. Simplified structure and reduced parameters can contribute to faster operational speed. The last one was selected and named as RTSD-Net. Comparing with YOLOv4 tiny, the accuracy of RTSD-Net is only reduced by 0.62% but the speed is increased by 25FPS, which is 25.93% higher than that of YOLOv4-tiny. Embedded system Jetson Nano was selected as the evaluation platform to evaluate the RTSD-Net’s performance for edge computing. The original Open Neural Network Exchange (ONNX) model was loaded on Jetson Nano and the speed of RTSD-Net was 13.1FPS, which is 19.0% higher than that of YOLOv4-tiny. After speeded up by TensorRT method, the transformed model reached 25.20fps, which is twice as fast as the ONNX model, and 15% faster than the YOLOv4-tiny model. After speeding up, the efficiency of RTSD-Net is enough for computer vision based strawberry detection and harvesting. In summary, the proposed RTSD-Net has good potential in smart strawberry harvesting machinery and the idea of redesigning neural structure and reducing parameters to speed up the detection rate of deep neural network is expected to have good application in edge computing.

Introduction

Machine learning is a hot topic in the field of artificial intelligence and it is the key technique for smart agricultural machinery. Deep learning is the forefront of the development of machine learning which has achieved excellent results in many ways like natural language processing and computer vision. Deep convolutional neural networks (DCNNs) have made significant progress in computer vision, which have greatly improved the performance of image classification, segmentation and detection. In terms of object detection, the accuracy of deep learning algorithms can be higher than visual recognition with incomparable efficiency. In the latest results, Deep learning algorithms have achieved 97% accuracy in the identification and positioning of rhizome features (Pound et al., 2018).

The current development of deep learning has two directions. One is to develop towards high precision, and the other is to develop towards high efficiency. Alexey AB (Bochkovskiy et al., 2020) proposed YOLOv4 with more prominent accuracy on the basis of YOLOv3, which improved data processing, network structure and loss function to achieve better detection accuracy. In addition to the accuracy improvement, the improvement in speed is also of great significance. Andrew (Howard et al., 2017) proposed the deep separable convolution network, and constructed the lightweight network MobileNet-v2 to greatly reduce the network parameters. In terms of the results, the accuracy of the network has declined to a certain extent while the detection speed has been greatly improved, which made it meet the requirements of specific scenarios. Based on YOLOv4 network, Alexey A.B proposed YOLOv4-tiny lightweight network, whose speed can reach 371 FPS when running on PC with GTX 1080 Ti. Besides, the model can also be deployed on embedded mobile devices, such as Jetson Nano or mobile smart phone. Lightweight network greatly promotes the deployment process of the model on edge computing devices.

Computer vision has experienced great achievement recently in agriculture automation and deep learning. For example, Chen (M. (Chen et al., 2021) obtained a detailed 3D orchard map combining eye-hand stereo vision and SLAM system. The candidate fruits are obtained by partial matching of contour information and probabilistic Hough transform, and then trained by SVM classifier. The experiment has good detection effect in most types of fruits (Lin et al., 2019). However, in smart agricultural machinery, many tasks need to be executed quickly on mobile devices like embedded systems, which makes researchers pay attention to the development of light weight deep neural network in edge computing. Deep learning algorithm has been used to identify crops, weeds and diseases in agriculture. (Y. (Chen et al., 2019) used UAV and RGB cameras to shoot strawberry fields, and used Faster-RCNN algorithm to detect the targets after segmentation. The number of strawberries and flowers was calculated to estimate the yield and accuracy rate has reached 84.1%. This research helps farmers to observe the growth cycle of strawberry fields in a more efficient way. (Gao et al., 2020) conducted research based on YOLOv3-tiny network to improve the speed of object detection and manage weeds in sugar beet field, which meets the requirements of speed and accuracy. Liu (Liu and Wang, 2020) proposes an early recognition method of tomato leaf spot based on MobileNetv2-YOLOv3 model to achieve a good balance between the accuracy and real-time detection of tomato gray leaf spot. Aguiar used deep learning algorithms such as MobileNets, Inception and lite YOLO to detect the trunk in real-time detection as an experimental performance benchmark between two devices is performed: NVIDIA’s Jetson Nano and Google’s USB Accelerator (Aguiar et al., 2020). Mazzia carried out the benchmark tests on two experimental devices, the Jetson Nano of NVIDIA and the USB Accelerator of Google for the real-time detection of apples in orchards and the evaluation of apple production (Mazzia et al., 2020).

Strawberries are of good taste and rich nutritional value, like multiple vitamins and amino acids(Proteggente et al., 2002). It’s well loved by the people all over the world. The worldwide consumption is expected to reach 11.5million tons. Among all producing countries China has the largest production and consumption. However, this industry is very labor-intensive, especially in harvesting, which makes the price of strawberries so high. Strawberry fruits should be harvested as soon as possible after maturity, but the problem is, the workload is so heavy that it is impossible to finish the harvest in such a short time. So, it is very meaningful to use automatic equipment to harvest strawberry. From this aspect, reducing labor force can really bring significant economic benefits

One key technique for smart harvesting is real-time object detection. How to improve the recognition speed of strawberries while ensuring the recognition accuracy of strawberries has become the focus of this work. Real-time detection algorithm is of great research significance in agriculture. In the single agricultural scene, the variety of object is far lower than some public datasets, such as COCO datasets. In the case of simple scene and fewer types of objects, how does reducing the complexity of network structure affect the accuracy and speed of object detection? In intelligent agricultural machinery, the improvement of detection speed is valuable. Although some researchers work on edge computing methods to deploy deep neural network on mobile platform, few of them works on the CNN structure to improve performance. The deep neural network consists of millions of parameters and sophisticated neural structures. Which parameter can be reduced and how is the effect on efficiency? This is a big issue in building DNN since some change may lead to bad effect.

In this research, we aim to find out a way to improve the performance of lightweight deep neural network and deploy it on embedded platform. And we aim to find out what can be done to the internal structure of YOLOv4-tiny so the speed can increase without compromising accuracy.

Section snippets

System setup

As the Fig. 1 shows, UAV was used for fast collection of data over the strawberry field. A total of 3 categories, strawberry, immature strawberry and flower were marked. Before the data set was put into the network, the data set needs preprocessing. On the network structure, we compared the detection results of 8 kinds of networks. 4 of them are YOLOv3, YOLOv3-tiny, YOLOv4, and YOLOv4-tiny, and the rest are the improved network structures. The corresponding speed was obtained by inference on PC

Accuracy

The YOLO algorithm shows good performance on the COCO data set. The mAP and F1 values ​​of YOLOv3 and YOLOv4 are higher than the YOLOv3-tiny and YOLOv4-tiny because YOLOv3 and YOLOv4 have more anchors to detect targets. We tested the mAP and F1 of 4 different YOLO models on the strawberry data set, as shown in Table 5. YOLOv3-tiny and YOLOv4-tiny are lightweight deep learning networks. The mAP of the YOLOv3-tiny model is significantly lower than the other 3 models. The mAP of YOLOv4-tiny is

Conclusion

Among the four existing YOLO networks, YOLOv4-tiny is the state-of-art deep neural network in object detection for edge computing as it is basically the same as YOLOv3 and YOLOv4 but it’s much faster than the two model. So, it’s selected as the basis of this research for further modifications to meet the demand of real-time strawberry detection. By gradually reducing the complexity of the network, the network parameters are reduced and the detection speed is improved. Among the 4 modified

Funding

This research was funded by the National Natural Science Foundation of China (NSFC), grant number: 61905219.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (25)

  • M. Chen et al.

    3D global mapping of large-scale unstructured orchard integrating eye-in-hand stereo vision and SLAM

    Comput. Electron. Agric.

    (2021)
  • A. Aguiar et al.

    Vineyard trunk detection using deep learning – An experimental device benchmark

    Comput. Electron. Agric.

    (2020)
  • Bochkovskiy, A., Wang, C.-Y., & Liao, H.-Y. M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv...
  • S. Cass

    Nvidia makes it easy to embed AI: The Jetson nano packs a lot of machine-learning power into DIY projects-[Hands on]

    IEEE Spectr.

    (2020)
  • Y. Chen et al.

    Strawberry Yield Prediction Based on a Deep Neural Network Using High-Resolution Aerial Orthoimages

    Remote Sens.

    (2019)
  • DeVries, T., & Taylor, G.W., 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint...
  • J. Gao et al.

    Deep convolutional neural networks for image-based Convolvulus sepium detection in sugar beet fields

    Plant Methods

    (2020)
  • K. He et al.

    Spatial pyramid pooling in deep convolutional networks for visual recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2015)
  • P. Henderson et al.
  • Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H., 2017. MobileNets:...
  • G. Lin et al.

    Fruit detection in natural environment using partial shape matching and probabilistic Hough transform

    Precis. Agric.

    (2019)
  • J. Liu et al.

    Early recognition of tomato gray leaf spot disease based on MobileNetv2-YOLOv3 model

    Plant Methods

    (2020)
  • Cited by (73)

    • Fruit sizing using AI: A review of methods and challenges

      2023, Postharvest Biology and Technology
    View all citing articles on Scopus
    View full text