Multi-class detection of kiwifruit flower and its distribution identification in orchard based on YOLOv5l and Euclidean distance

Highlights

• Selecting suitable flower for pollination based on flowers detection and distribution.

• Kiwifruit flowers were labeled into 10 classes based on flower phenology.

• Flower cluster and branch junction were applied for obtaining flower distribution.

• YOLOv5l reached mAP of 91.60 % on multi-class objects detection in 15.50 ms per image.

• The method reached total MA of 93.30 % with processing speed of 112.46 ms per image.

Abstract

Asynchrony of kiwifruit flowering time results in different flower phenological stages in canopy at the same time. Pollination quality of flowers is influenced by their phenological stages, while their distributions determine fruit distributions and influence kiwifruit quality and yield. Thus, it’s necessary to find suitable flowers to be pollinated based on flower phenology and its distribution. However, influences of flower phenology and flower distribution were not considered in most previous studies about robotic pollination of kiwifruit, where pollination of all open flowers was indiscriminate. Therefore, a method was proposed for multi-class detection of kiwifruit flower and its distribution identification in orchard, which was based on You Only Look Once version 5 large (YOLOv5l) and Euclidean distance. According to kiwifruit flower phenology, kiwifruit flowers were classified into 10 classes to find suitable flowers for pollination, while flower cluster and branch junction were divided into 4 classes for obtaining flower distributions. All classes were manually labeled by rectangular bounding boxes for training and testing. Considering high detection accuracy requirements with small model size, YOLOv5l was applied to do transfer learning for multi-class detection of kiwifruit flower. Then, pixels coordinate of multi-class objects and their corresponding Euclidean distances could be gained. Finally, flower distributions in canopy were obtained by matching method. Total mean Average Precision (mAP) was 91.60 % in YOLOv5l, while the mAP of multi-class flower (10 classes) was 93.23 %, which was 5.70 % higher than that of the other 4 classes. Matching accuracy (MA) of flowers matching flower clusters was up to 97.60 %. Moreover, MA of flower cluster matching branch junction (96.20 %) and total MA (93.30 %) increased by 1.20 % and 1.00 % based on improved matching method, respectively. Total processing speed of multi-class flower detection and its distribution identification was 112.46 ms per image including 15.50 ms for image detection by YOLOv5l. Results showed that multi-class kiwifruit flowers and relative flower distributions could be fast and accurately obtained for further selecting suitable flowers for robotic pollination.

Introduction

Pollination in suitable flower phenology is critical to achieving satisfactory kiwifruit production and quality. Kiwifruit size is directly related to seed number, i.e., to the number of fertilized ovaries, which depend on pollination (Gonzalez et al., 1995). However, kiwifruit is more difficult than other species on pollination, since not only it’s dioecious without synchrony of flowering time, but also its flowers lacking nectar are not enough attractive to pollinators (Castro et al., 2021). Artificial pollination overcame difficulty and became a critical technique to increase kiwifruit quality (Lim et al., 2020). Nevertheless, artificial pollination does not always reach maximal efficiency, which depends on female flower phenological stages at pollination, a pollination system (dry or liquid), and pollination methods (Gianni and Vania, 2018). Flower phenology has been considered during artificial pollination by workers, which was hardly to implement automatically in robotic pollination. Therefore, it is necessary to develop robotic pollination based on flower phenology, which can not only replace manual pollination to save high labor costs but also improve pollination quality of kiwifruit flowers to increase kiwifruit quality and size.

Unbalanced flower distributions may cause unbalanced crop load and nutrient delivery, which can be improved by flower and fruit thinning. Early flowers (usually in middle part of fruit branch) had larger ovaries with more locules and ovules than late flowers (in base portion or top of fruit branch) on the same vine and they produced larger fruit (McPherson et al., 2001). Comparisons between early and late flowers may be a result of competition for resources within fruit branch and positional effects (Smith et al., 1994). Although flower thinning or fruit thinning was usually taken to adjust flower or fruit distributions (fruit distributions mainly depending on flower distributions), flower thinning was more effective in enhancing fruit size and weight as compared to fruitlet thinning (Cangi and Atalay, 2018, Wu et al., 2022). Besides, without pollination or invalid pollination, ovary of kiwifruit flower had drop phenomenon (Thakur and Chandel, 2004). Therefore, some suitable flowers (mainly in middle part of fruit branch) could be chosen to be pollinated based on their distributions to replace flower thinning and decrease work of fruit thinning.

Kiwifruit flowers can be divided into multi-class flowers based on their flower phenology to select suitable flowers for pollination. Researchers did many relative studies on classification of kiwifruit flowers. Salinero et al. (2009) proposed BBCH (named after its developing institutes Biologische Bundesanstalt, Bundessortenamt, und CHemische Industrie) scale of kiwifruit plant, which divided flowering phenology into closed flower, white petals, ocher petals, early petal fall, and petal fall. Although the BBCH scale of kiwifruit plant provided information on flower phenology, relations between phenological flowering stages and pollination were vague. According to phenological flowering stages of kiwifruit, Gianni and Vania (2018) divided flowers into 5 classes before pollination, i.e., closed flower, white petals, ocher petals, early petals fall, and petals fall. They found out that dry pollination was better made with pure pollen before petal fall stage, while liquid pollination was not later than early petal fall stage. Previous studies about classification of kiwifruit flowers were just used for optimal orchard management such as artificial pollination and flower thinning (mainly relying on humans). However, with rapid development of agricultural robot, it’s necessary to combine knowledge of agronomic pollination with robot for precision pollination. Thus, multi-classification of kiwifruit flowers was essential for robotic pollination based on flower phenology.

It is very challenging to identify flower phenology and flower distributions due to complex kiwifruit orchard environments (such as overlapping flowering phenology and occluding situations), which may be solved by image processing algorithms. Most previous studies focused on identification of multi-class fruit for harvesting or multi-class flower for flower thinning, where image processing algorithms were mainly divided into traditional image processing technology and deep learning methods. Traditional approaches mainly focused on features such as color and texture using complex algorithms with many fixed thresholds, which probably produced non-robust and inaccurate results in complex environments of orchard (Fu et al., 2021, Tian et al., 2020, Majeed et al., 2020). With rapid development of deep learning methods, a considerable number of researchers applied them for classification tasks in agricultural field, which conquered limitations of traditional image processing technology (Dias et al., 2018, Wu et al., 2021, Zhang et al., 2020, Zhang et al., 2021). Gao et al. (2020) proposed a multi-class apple (non-occluded, leaf-occluded, branch/wire-occluded, and fruit-occluded fruit) detection method based on Faster Region-Convolutional Neural Network (R-CNN) with Visual Geometry Group with 16 layers (VGG16), which obtained mean Average Precision (mAP) of 87.85 % and detection speed of 241.0 ms per image with 1920 × 1080 pixels. Although Faster R-CNN (classic of two-stage networks) achieved high accuracy in complex orchard environments, its detection speed was slow. Therefore, object detection models need to be selected in terms of both accuracy and speed.

You Only Look Once (YOLO), as a one-stage network, showed high potential in speed while achieving high accuracy in agricultural field. Suo et al. (2021) employed YOLOv4 for identification of five-classes kiwifruit (fruit not occluded, fruit occluded by leaves, fruit occluded by other fruits, fruit occluded by branches, fruit occluded by wires), which achieved mAP of 91.9 % and detection speed of 25.5 ms per image with 2,352 × 1,568 pixels. Jocher et al. (2020) proposed YOLOv5 in PyTorch framework, the latest version of YOLO family, which was divided into many versions based on model complexity, namely YOLOv5s, YOLOv5m, YOLOv5l, YOLOv5x, and so on. Therefore, compared with YOLOv3 and YOLOv4, YOLOv5 had a more flexible network structure to choose appropriate one more conveniently based on different tasks (Bochkovskiy et al., 2020, Kuznetsova et al., 2020, Lin et al., 2021, Redmon and Farhadi, 2018). Then, in another field of agriculture, Wang et al. (2021) proposed a method for estimation of apple flower phenology distribution based on YOLOv5l to gain timing of flower thinning, which acquired classification accuracy up to 92 % and inference time of 16 ms per image with 4,196 × 2,160 pixels. YOLO models, especially YOLOv5, achieved considerably satisfactory results on classification tasks in orchard. Therefore, among YOLOv5 models, YOLOv5l was selected for this study since this model attained high detection accuracy and fast detection speed.

This study aimed to develop a multi-class detection of kiwifruit flower and its distribution identification method based on YOLOv5l and Euclidean distance to select suitable kiwifruit flowers for robotic pollination. This work is organized as follows. Section 2 will describe the design of this proposed method; Section 3 will present the obtained results of multi-class kiwifruit flower detection and matching among multi-class objects in canopy, and discuss the relevant issues; Section4 will present the conclusions and prospects acquired from this study.