Apple detection in nighttime tree images using the geometry of light patches around highlights
Introduction
Localization of fruit on trees remains a challenging issue, which has potential applications ranging from early estimation of fruit load which could be used to manage the orchard (thinning operations, irrigation, etc.) to localization of mature fruit for robotic harvesting or yield estimation. A first review devoted to fruit detection was published in 2000 by Jimenez et al. (2000a). More recently, Kapach et al. (2012) surveyed the advances and challenges still faced by computer vision for robotic harvesting. Despite more than 30 years of research, the unstructured, uncontrolled, cluttered outdoor environment which is typical of agricultural operations still presents many challenges for computer vision systems. Due to the complexity of the task various approaches have been investigated, ranging from a simple RGB camera to systems which provide three dimensional information (such as laser scanners, multi-views systems, structured light, time-of-flight, e.g. Jiménez et al., 1999, Jiménez et al., 2000b, Rakun et al., 2011, Barnea and Ben-Shahar, 2014), hyperspectral imaging (e.g. Annamalai and Lee, 2003, Safren et al., 2007) or thermal imaging (e.g. Stajnko et al., 2004, Bulanon et al., 2008, Wachs et al., 2010). The advantages and limitations of the various approaches are discussed in Kapach et al. (2012). The use of a standard RGB camera has obvious advantages in terms of cost and ease of operation, and such a simple configuration is still the focus of numerous studies (e.g. Kurtulmus et al., 2011, Linker et al., 2012, Payne et al., 2013, Zhou et al., 2012, Bansal et al., 2013, Kelman and Linker, 2014). Most studies which attempted to detect green fruit within green foliage with a single RGB camera under natural illumination concluded that the uncontrolled illumination made it very difficult to achieve robust and reliable results. In order to overcome this, several studies have investigated the use of nighttime imaging under artificial illumination (e.g. Sites and Delwiche, 1988, Payne et al., 2014, Cohen et al., 2014). These studies considered the more homogenous illumination of the scene to be the main advantage of this technique. In the present study we investigated the usefulness of another feature of nighttime imaging, namely the fact that in such images convex surfaces exhibit a “bright spot” due to specular reflection (so-called specular highlights). Specular highlights may also exist in daytime images, and Mairon and Ben-Shahar (2014) recently reported the use of this feature for detecting sweet peppers in greenhouses. When a strong source of artificial light roughly aligned with the camera is used, as in nighttime imaging, these highlights are more pronounced. Although such highlights are usually considered a nuisance and numerous studies have been devoted to removing them (e.g. Tan et al., 2006, Blanc-Talon et al., 2009), Font et al. (2014) recently showed that these highlights could be used to estimate the number of grapes in nighttime images. A similar approach is followed in the present study. However, whereas Font et al. (2014) were able to use simple filtering techniques to mask the background (and hence most of the highlights not due to grapes), such background removal is not possible for typical tree images. In order to distinguish between highlights on apples and highlights on leaves, branches or other reflecting surface, we developed a procedure that relies on the spatial variation of the light intensity around the highlight.
The present work is part of a project whose ultimate goal is the development of a vision system for estimating orchard yield. As such, and contrary to vision systems for harvesting robots, the emphasis is not on exact localization of the fruits but rather on accurate estimation of the number of fruits in the images. Also, since each image covers a large portion of the tree, the apparent size of the fruit is much smaller than in images acquired for robotic harvesting.
Section snippets
Datasets
The study involved two datasets of images captured in ‘Golden Delicious’ orchards at the Matityahu Research Station located in Northern Israel. The acquisition of both datasets was started one hour after sunset on cloudless nights. In order to create datasets that included trees with both high and low fruit loads, images were acquired in two areas known to produce very different yields. In each area images were acquired in two rows, labeled H1 and H2, and L1 and L2, respectively. The planting
Results and discussion
The procedure was initially developed and tuned using the 2012 dataset. Twenty images from this dataset were selected randomly to form the calibration set, based on which the parameter values listed in Appendix A were obtained. Unless otherwise stated, these values were used to obtain all the results presented below. The 2013 dataset differed from the 2012 one in terms of the apparent size of the apples (average radius of ∼60 pixels compared to ∼100 pixels) and a much larger range of fruit
Conclusions
Estimating the number of green fruits in tree images remains a challenging issue. In the present work we developed an approach that relies on the specular highlight (“bright spot”) which appears on a convex surface when the light source and camera are roughly aligned, such as when using artificial illumination. The main advantage of this approach is that it is not sensitive to fruit coloring. The procedure includes only a small number of parameters which reflect mostly the texture and apparent
Acknowledgements
This study was supported by a joint grant from the Center for Absorption in Science of the Israel Ministry of Immigrant Absorption and the Committee for Planning and Budgeting of the Council for Higher Education in Israel under the frame work of the KAMEA Program.
References (24)
- et al.
Study on temporal variation in citrus canopy using thermal imaging for citrus fruit detection
Biosyst. Eng.
(2008) - et al.
Counting red grapes in vineyards by detecting specular spherical reflection peaks in RGB images obtained at night with artificial illumination
Comput. Electron. Agric.
(2014) - et al.
Automatic fruit recognition: a survey and new results using range-attenuation images
Pattern Recogn.
(1999) - et al.
Vision-based localization of mature apples in trees using convexity
Biosyst. Eng.
(2014) - et al.
Green citrus detection using ‘eigenfruit’, color and circular Gabor texture features under natural outdoor conditions
Comput. Electron. Agric.
(2011) - et al.
Determination of the number of green apples in RGB images recorded in orchards
Comput. Electron. Agric.
(2012) - et al.
Estimation of mango crop yield using image analysis–segmentation method
Comput. Electron. Agric.
(2013) - et al.
Estimating mango crop yield using image analysis using fruit at ‘stone hardening’ stage and night time imaging
Comput. Electron. Agric.
(2014) - et al.
Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry
Comput. Electron. Agric.
(2011) - et al.
Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging
Comput. Electron. Agric.
(2004)