Grapevine buds detection and localization in 3D space based on Structure from Motion and 2D image classification

Highlights

• Grapevine bud 3D localization in natural field conditions.

• First steps toward high throughput plant structuring.

• Multi-view 3D reconstruction workflow for high precision localization in noisy conditions.

Abstract

In viticulture, there are several applications where 3D bud detection and localization in vineyards is a necessary task susceptible to automation: measurement of sunlight exposure, autonomous pruning, bud counting, type-of-bud classification, bud geometric characterization, internode length, and bud development stage. This paper presents a workflow to achieve quality 3D localizations of grapevine buds based on well-known computer vision and machine learning algorithms when provided with images captured in natural field conditions (i.e., natural sunlight and the addition of no artificial elements), during the winter season and using a mobile phone RGB camera. Our pipeline combines the Oriented FAST and Rotated BRIEF (ORB) for keypoint detection, a Fast Local Descriptor for Dense Matching (DAISY) for describing the keypoint, and the Fast Approximate Nearest Neighbor (FLANN) technique for matching keypoints, with the Structure from Motion multi-view scheme for generating consistent 3D point clouds. Next, it uses a 2D scanning window classifier based on Bag of Features and Support Vectors Machine for classification of 3D points in the cloud. Finally, the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) for 3D bud localization is applied. Our approach resulted in a maximum precision of 1.0 (i.e., no false detections), a maximum recall of 0.45 (i.e. 45% of the buds detected), and a localization error within the range of 259–554 pixels (corresponding to approximately 3 bud diameters, or 1.5 cm) when evaluated over the whole range of user-given parameters of workflow components.

Introduction

In this work, we present an approach for the efficient 3D detection and localization of grapevine buds. 3D models were reconstructed from multiple images captured during the winter season in natural field conditions (i.e., natural sunlight and the addition of no artificial elements) using a mobile phone RGB camera.

Grapevine buds were recognized early in viticulture history as one of the most important parts of the plant, mainly because they contain the whole plant productive capacity, from which all sprouts, leaves, bunches, and tendrils grow. In particular, bud bunch fertility, a.k.a. fruitfulness, is of particular interest, as it has a direct impact on the main goal of vine production, that is, to increase productivity without affecting fruit quality. It has been shown that bud fruitfulness depends on the amount of sunlight exposure of buds during the period starting at bud initiation in early spring throughout its development stage up to 30 days after bloom [[15], [21], [11], [25], [35], [27]]. Shading conditions during this period strongly depend on what we call shading structure, consisting in the localization and geometric characterization of those parts of the plant that occlude sunlight, mainly the leaves and bunches that grow after bloom. In addition, sunlight exposure can be used by growers to influence the productivity of the next period by choosing those buds that received the most sunlight exposure. In practice, this happens by deciding pruning procedures late in the winter [23]. There is a balance, however, as unpruned buds will produce vegetation, shading the newly initiated buds, and therefore, affecting the productivity of the next period. The decision of optimal pruning is, therefore, a complex task that must be carefully balanced between: (i) productivity maximization of the starting period determined by buds with maximum sun exposure, and (ii) productivity maximization of the following period determined by the shading conditions resulting from the green vegetation growing from those buds.

A solution to the first issue requires measuring the sun exposure of individual buds at regular intervals from initiation to 30 days after bloom and then recovering this value for each bud months later during winter pruning. Sunlight exposure has been measured so far through manual positioning of radiation sensors [25]. These manual procedures, however, are far from efficient for the massive measuring of sunlight exposure of individual plants, not to mention of individual buds. Our work aims to partially fulfill the need for an efficient method for measuring and recording the sunlight exposure of individual buds. The general rationale behind our approach is that it is possible to compute the sunlight exposure of a bud with high-precision when the precise 3D localization of the bud, the shading structure around it, the geo-positioning of the field, and the dates of interest are fed to a sun radiation model [[29], [8]]. It is an ambitious goal, attended partially by the present work that provides a solution to the 3D localization of winter buds. Future work, however, will have to solve the problem of producing the shading model. This could be done by localizing buds from initiation till the end of summer, and then by identifying buds between consecutive 3D modelizations to allow the recording of long-term sun exposure. A solution to the second issue requires a thorough understanding of which summer shading structures result from different winter pruning procedures and trellis systems [[11], [14]]. This demands measuring the shading structure, a procedure which is currently unavailable.

Simulations are a possibility for partially overcoming the inability to reconstruct the shading structure, necessary for solving both issues. There is a line of research that studies different procedures for producing simulated whole plant shading structures, including the canopy and bunches [[13], [16]]. They typically require plant architecture and bud localization as input. However, bud localization information, being inexistent, is provided by randomly simulating their position. Our work provides a solution to the latter, while [26] is one of the many studies that provide a solution to the former. Despite being a simulated model, the shading structure has the potential to produce invaluable—and to this day inexistent—information on the (simulated) long-term sun exposure of large bud samples, including months with a fully grown canopy. In particular, with plant architecture before the winter pruning, it is possible to simulate the backward shading structure of the previous spring as well as different forward shading structures resulting from different pruning treatments.

Finally, we note that both issues require an autonomous system for executing pruning. Historically, pruning procedures have been simplified to be accessible for humans. However, this may change with the extra information provided by 3D modeling, namely, the identification of fruitful buds and predictions of next-period's shading structures. With this information, the resulting optimal pruning may be too sophisticated to be amenable for human execution, requiring autonomous pruning systems.

In addition to measuring sunlight exposure and guiding autonomous pruning, bud localization is also required as part of the measuring processes of other variables of interest in viticulture. These are bud count, type-of-bud classification, bud geometric characterization, internode length, and bud development stage. Their values at any location are of importance to agronomists for deciding on possible treatments (e.g., the application of fertilizers, canopy pruning), or for predicting plant productivity. Observation and measurement of crop variables is a fundamental task that offers the agronomist information about crop state, providing the means for informed decision-making of what treatments must be applied in order to maximize productivity and crop quality. At present, these variables are measured through direct or indirect human visual inspection, whose elevated cost often results in the measurement of only a small sample of all cases. When data are scarce, even powerful statistical techniques may still result in high uncertainty in the decision-making process, motivating the introduction of improved sensing procedures. Locating buds is a necessary task to conduct a proper measurement of the above variables. However, 2D localization is sufficient for all variables with the exception of internode length, for which 3D localization of two consecutive buds in a cane is necessary to avoid perspective errors. Still, automatic, high-throughput measurement of these variables would come with no extra cost with an autonomous 3D localization system in place.

There are many computational approaches to aid viticulture, including detecting grapes and bunches, estimating grape size and weight, estimating production and foliar area indexes, phenotyping, and autonomous selective pulverization [[19], [30], [6], [12], [2], [31]]. For a more extensive review, see [37].

Specifically concerning the detection of grapevine buds, there are two recent studies (in 2D only) that address the problem of grapevine bud detection [[38], [12]]. The first one presents a grapevine bud detection algorithm designed specifically to establish the groundwork for a future autonomous pruning system in the winter season (with no leaves left that may occlude the vision and operation of the cutting mechanism). Bud detection is performed from RGB images (the image resolution in this study is unknown). Furthermore, on top of this assumption, images are captured indoors with an industrial CCD camera with controlled background and lighting conditions. To discriminate between plant and background pixels, the authors apply a simple threshold resulting in a binary image to obtain a wire skeleton of the plant. Under the assumption that bud morphology is similar to that of the corners, they apply Harris’ algorithm [9] to the skeleton image for detecting those corners. This process produces a recall of 0.702, i.e., 70.2% of buds detected. Although some improvements are suggested by the authors, the most striking limitations of this work are the need for images captured under controlled indoors conditions and the fact that the resulting localizations are in 2D. A second work for bud detection is presented by Herzog et al. [12]. This work introduces three methods of bud detection. The best results are obtained with the semi-automatic method that requires human intervention for validating the quality of the results. Detection is based on 3456 × 2304 RGB images, where the scene is altered with an artificial black background, producing a recall of 0.94. The authors argue that this recall is enough to satisfy the phenotyping of plants. However, as the authors themselves point out, these good results are mainly explained by the particular color and morphology of the buds, captured when bud sprouts are visibly green and their average size is around 2 cm (compared to a typical 5 mm diameter of a dormant bud) which makes it easier to discriminate them visually from other plant components. Although these works represent important advancements in specific bud detection applications, they suffer from some of the following limitations: (i) the use of an artificial background, (ii) controlled indoors luminosity, (iii) the need for human intervention, (iv) the detection of buds in an advanced stage of development, and (v) detection is in 2D.

Dey et al. [5] introduced a pipeline for recovering the 3D structure of the grapevine plant in the spring–summer season (i.e., with leaves and fruits) from a 3D point cloud. This 3D point cloud visually represents the surface parts of the environment, where each point is represented by a tuple containing the 3D position in world coordinates (x, y, z). Cloud reconstruction is obtained with the algorithm proposed by Snavely et al. [28]. Afterwards, the cloud is classified into leaves, branches, and fruits by means of a supervised classification algorithm that uses shape and color features. The experiments show an accuracy of 0.98 for grapes before maturation (still green) and 0.96 for fully ripe grapes (color change), where accuracy corresponds to the proportion of all observations (both grapes and background) that were correctly classified. Despite the similarities with our work, their work classifies grapes and ours classifies buds, making it hard to compare them. This is mainly due to the geometrical nature of the features they use that one would expect to work better for close-to-spherical shapes such as that of grapes, but which may work poorly for buds that present a highly irregular shape.