Apple detection in nighttime tree images using the geometry of light patches around highlights

https://doi.org/10.1016/j.compag.2015.04.005Get rights and content

Highlights

  • Highlights (“bright spots”) were used to detect apples in nighttime images.

  • The spatial distribution of the light around the highlights was used to reject false positives.

  • The procedure was tested with two dataset containing over 360 images.

  • The total number of fruit estimated by the procedure was within 10% of visual count.

Abstract

Detection of fruit in tree images has been the focus of numerous studies. Although most studies considered approaches based primarily on color analysis, the major drawback of such approaches is that the fruit apparent color depends not only on variety or physiological stage but also on illumination, which is inherently non-uniform within the canopy, even if artificial lighting is used. In the present work we developed a novel approach to detect apples in nighttime images by analyzing the spatial distribution of the light around highlights (“bright spots”). The approach is based on the observation that, under the artificial illumination used, apples exhibit strong specular reflection so that a small, but very bright, spot is visible on almost all apples. Each of these highlights serves as the center of a region of interest and is the seed of the investigated light patch. This patch is initially very small but its size is increased iteratively by annexing pixels with predefined decreasing gray level intensities. The evolution of the patch geometry is used to determine whether it corresponds to an apple. The approach was tested with two datasets containing over 360 images (close to 13,000 apples) acquired in the same ‘Golden Delicious’ orchard in July 2012 and August 2013. Twenty images from the 2012 dataset were randomly selected to develop and calibrate the procedure. The results of these 20 images were used to establish a linear relationship between the number of detected objects and the actual number of apples visible in the images (R2  0.75). Applying the calibrated procedure to the remaining images of this dataset led to an estimate of 6739 apples compared to a visual count of 6195 apples (∼9% overestimate). Analysis of the 2013 dataset, in which the apparent size of the apples was smaller, required only adjustment of the two parameters related to apple size. Following this adjustment, 12 images were randomly selected to determine the relationship between the number of detected objects and the actual number of apples (R2  0.74). Using this relationship, the estimated number of apples was 6687, compared to the visual count of 6713 fruits.

Introduction

Localization of fruit on trees remains a challenging issue, which has potential applications ranging from early estimation of fruit load which could be used to manage the orchard (thinning operations, irrigation, etc.) to localization of mature fruit for robotic harvesting or yield estimation. A first review devoted to fruit detection was published in 2000 by Jimenez et al. (2000a). More recently, Kapach et al. (2012) surveyed the advances and challenges still faced by computer vision for robotic harvesting. Despite more than 30 years of research, the unstructured, uncontrolled, cluttered outdoor environment which is typical of agricultural operations still presents many challenges for computer vision systems. Due to the complexity of the task various approaches have been investigated, ranging from a simple RGB camera to systems which provide three dimensional information (such as laser scanners, multi-views systems, structured light, time-of-flight, e.g. Jiménez et al., 1999, Jiménez et al., 2000b, Rakun et al., 2011, Barnea and Ben-Shahar, 2014), hyperspectral imaging (e.g. Annamalai and Lee, 2003, Safren et al., 2007) or thermal imaging (e.g. Stajnko et al., 2004, Bulanon et al., 2008, Wachs et al., 2010). The advantages and limitations of the various approaches are discussed in Kapach et al. (2012). The use of a standard RGB camera has obvious advantages in terms of cost and ease of operation, and such a simple configuration is still the focus of numerous studies (e.g. Kurtulmus et al., 2011, Linker et al., 2012, Payne et al., 2013, Zhou et al., 2012, Bansal et al., 2013, Kelman and Linker, 2014). Most studies which attempted to detect green fruit within green foliage with a single RGB camera under natural illumination concluded that the uncontrolled illumination made it very difficult to achieve robust and reliable results. In order to overcome this, several studies have investigated the use of nighttime imaging under artificial illumination (e.g. Sites and Delwiche, 1988, Payne et al., 2014, Cohen et al., 2014). These studies considered the more homogenous illumination of the scene to be the main advantage of this technique. In the present study we investigated the usefulness of another feature of nighttime imaging, namely the fact that in such images convex surfaces exhibit a “bright spot” due to specular reflection (so-called specular highlights). Specular highlights may also exist in daytime images, and Mairon and Ben-Shahar (2014) recently reported the use of this feature for detecting sweet peppers in greenhouses. When a strong source of artificial light roughly aligned with the camera is used, as in nighttime imaging, these highlights are more pronounced. Although such highlights are usually considered a nuisance and numerous studies have been devoted to removing them (e.g. Tan et al., 2006, Blanc-Talon et al., 2009), Font et al. (2014) recently showed that these highlights could be used to estimate the number of grapes in nighttime images. A similar approach is followed in the present study. However, whereas Font et al. (2014) were able to use simple filtering techniques to mask the background (and hence most of the highlights not due to grapes), such background removal is not possible for typical tree images. In order to distinguish between highlights on apples and highlights on leaves, branches or other reflecting surface, we developed a procedure that relies on the spatial variation of the light intensity around the highlight.

The present work is part of a project whose ultimate goal is the development of a vision system for estimating orchard yield. As such, and contrary to vision systems for harvesting robots, the emphasis is not on exact localization of the fruits but rather on accurate estimation of the number of fruits in the images. Also, since each image covers a large portion of the tree, the apparent size of the fruit is much smaller than in images acquired for robotic harvesting.

Section snippets

Datasets

The study involved two datasets of images captured in ‘Golden Delicious’ orchards at the Matityahu Research Station located in Northern Israel. The acquisition of both datasets was started one hour after sunset on cloudless nights. In order to create datasets that included trees with both high and low fruit loads, images were acquired in two areas known to produce very different yields. In each area images were acquired in two rows, labeled H1 and H2, and L1 and L2, respectively. The planting

Results and discussion

The procedure was initially developed and tuned using the 2012 dataset. Twenty images from this dataset were selected randomly to form the calibration set, based on which the parameter values listed in Appendix A were obtained. Unless otherwise stated, these values were used to obtain all the results presented below. The 2013 dataset differed from the 2012 one in terms of the apparent size of the apples (average radius of ∼60 pixels compared to ∼100 pixels) and a much larger range of fruit

Conclusions

Estimating the number of green fruits in tree images remains a challenging issue. In the present work we developed an approach that relies on the specular highlight (“bright spot”) which appears on a convex surface when the light source and camera are roughly aligned, such as when using artificial illumination. The main advantage of this approach is that it is not sensitive to fruit coloring. The procedure includes only a small number of parameters which reflect mostly the texture and apparent

Acknowledgements

This study was supported by a joint grant from the Center for Absorption in Science of the Israel Ministry of Immigrant Absorption and the Committee for Planning and Budgeting of the Council for Higher Education in Israel under the frame work of the KAMEA Program.

References (24)

  • Annamalai, P., Lee, W.S., 2003. Citrus yield mapping system using machine vision. ASAE Paper...
  • R. Bansal et al.

    Green citrus detection using fast Fourier transform (FFT) leakage

    Precision Agric.

    (2013)
  • Cited by (0)

    View full text