Image Based Estimation of Fruit Phytopathogenic Lesions Area

Abstract

A method was developed to measure the surface area of walnut fruit phytopathogenic lesions from images acquired with a basic calibration target. The fruit is modelled by a spheroid, established from the 2D view ellipse using an iterative process. The method was tested with images of colour circular marks placed on a wooden spheroid. It proved effective in the estimation of the spheroid semi-diameters (average relative errors of 0.8% and 1.0%), spheroid surface (1.77%) and volume (2.71%). The computation of the colour mark surface area was within the expected error, considering the image resolution (up to about 4%), for 22 out of 28 images tested.

1 Introduction

The early detection of plant diseases caused by phytopathogenic bacteria is of upmost importance to assist the phytosanitary regulatory bodies in the implementation of timely and efficient control measures [2].

Xanthomonas arboricola pv. juglandis ($X_{aj}$) is the etiological agent of the “walnut (Juglandis regia L.) bacteria blight” (WBB) affecting leaves, and the “brown apical necrosis” (BAN) resulting in premature fruit drop. The presumptive diagnosis of these walnut diseases, which affect severely the walnut production worldwide [7], is made by the observation of symptomatic lesions characterized by necrotic spots or patches on walnut leaves and fruits. The presence of dark brown spot lesions is an initial sign of a possible infection by $X_{aj}$, which can be further confirmed by their development over time, both in terms of average area covered and incidence within a walnut symptomatic tree or an orchard. Therefore, the identification of these lesions is an immediate indication for both phytosanitary inspectors and producers, to implement suitable phytosanitary practices and to make comparative assessments between samples at different time points to infer the aggressiveness of the disease, and gather informative data for epidemiological records. However, in order to analyze the temporal development of the lesions, that may help to determine the virulence of a particular $X_{aj}$ strain or the resistance of a specific walnut host cultivar, quantitative measurements are required.

There are a number of methods for the computation of fruit area, but most of these methods are not suitable to quantitatively evaluate the extension of the lesions and consequently the disease aggressiveness. Several papers present methodologies for the evaluation of fruit shape parameters, such as surface area or volume [1, 3, 8, 9]. However, these approaches compute the full surface area, and not a specific sub-region. Furthermore, most of these methods involve stereoscopic cameras [3] and well controlled illumination [6, 8], which is not adequate for in field measurements.

This work is part of the EVOXANT project - Evolution of Xanthomonas arboricola species complex beyond culturability. One of its objectives is to develop a low cost, easy to use system for the acquisition of lesion area data from fruits in the field by farmers or phytosanitary technicians. This paper presents the system that is being developed for that purpose, focusing on the geometric aspects of the problem. The data used is collected by a simple camera (such as those available in smartphones and tablets) and a basic calibration target. The paper includes three additional chapters to this introduction. Section 2 presents the methodology, including imaging geometry, 3D fruit modeling and error estimation; Sect. 3 presents an experimental evaluation using color marks of fixed size over a wooden spheroid; and Sect. 4 highlights the main conclusions.

Fig. 1.

(a) Walnut fruit acquired with A4 calibration target; (b) 2D ellipse modelled for (a); (c) wooden spheroid in A4 target; (d) 2D ellipse modelled for (c). (Color figure online)

2 Methodology

2.1 Image Acquisition and Pre-processing

The images are acquired with the object of interest placed over a calibration target, which was designed to assist in the spacial referencing. The calibration target consists of a set of 4 marks (red crosses) on a white paper, forming a rectangle, with the distance between their centres known. It can be created with a standard printer, with matte paper of any format. The only requirements are that the marks are red, they form a rectangle, and the rectangle length and width are known. For in-field use, it is preferable to use a rigid material, for example wood or plastic. The images can be acquired with any standard camera, preferably with the rectangle formed by the red marks roughly aligned with the image grid and covering most of the image. As an example, Fig. 1(a) shows an image of a walnut fruit and Fig. 1(c) a wooden spheroid, both acquired with an A4 calibration target. The test images used in this work were obtained with an Apple iPAD mini 5MP camera ($2592\times 1936$ pixels).

A number of pre-processing tasks are performed in order to identify and evaluate the object or sub-region of interest. The initial step is to identify the calibration marks, which is done by thresholding in the HSV colour domain, followed by a sequence of morphological operations [5]. The image coordinates of the mark centres are used to compute 6 estimates of the average image resolution at the calibration plane ($R_{0}$ in pixel/mm). The region of interest is established as the polygon defined by the 4 marks, removing a margin so that the red marks are not included. A global thresholding is applied to the HSV image, with further morphological operations performed to segment the single largest object present in the region of interest [5].

Considering that the object of interest is adequately segmented, the next step is to identify a sub-region within the object. It is a 2-class classification problem, which can also be considered as a binary segmentation of an RGB image. In the case of a walnut fruit, the background is green and the foreground brown (lesion). An alternative case is considered here, with a spheroid with light brown background and a blue circular mark, as illustrated in Fig. 1(c). The choice of colours was made to simplify the segmentation task, as the paper focus on the geometric aspects of the problem, particularly the measurement of a 3D object size using a single image.

Fig. 2.

Schematic representation of near vertical imaging geometry.

2.2 Imaging Geometry

The reconstruction of the position and shape of objects from photographs requires a pair of stereoscopic images [4]. However, a single photograph might be enough for a plane object, provided that the distance between the projection centre and the plane along the principal axis ($D_{0}$) is known. As the photogrammetric equations used for the central projection of a plane requires several inner orientation (camera) parameters [4], not available for standard cameras, a simplified approach is considered for near-vertical images. It requires only 1 camera parameter - the principal distance, also referred to as or focal length (f), or the camera field of view ($\beta$). A schematic representation of the imaging geometry is presented in Fig. 2(a). The image scale in the reference plane ($S_{0}$) is computed as $S_{0}=f/D_{0}$, or alternatively as $S_{0}=l/L$. For an object of height H, placed over the reference plane, a new scale factor ($S_{1}$) needs to be computed, using (1). It requires the knowledge of H, $S_{0}$, and $D_{0}$ or f.

$$\begin{aligned} S_{1}=\frac{f}{D_{1}}=\frac{f}{D_{0}-H}=\frac{S_{0}D_{0}}{D_{0}-H} \end{aligned}$$

(1)

$$\begin{aligned} R_{1}=\frac{L_{0}R_{0}}{L_{1}}=\frac{D_{0}R_{0}}{D_{1}}= \frac{D_{0}R_{0}}{D_{0}-H}=\frac{R_{0}L_{0}}{L_{0}-2Htan(\beta /2)} \end{aligned}$$

(2)

A perhaps more relevant parameter than S (dimensionless) is the image resolution (R, in pixel/mm). An estimate for the calibration plane ($R_{0}$) is computed from the reference marks. For an object of some height (H), the image resolution ($R_{1}$) is computed by (2), illustrated in Fig. 2(b). The parameter $\beta$ is assumed constant for a given camera. If unknown, it can be estimated by acquiring an image and measuring $D_{0}$. The only unknown is thus the object height (H), considering that both $R_{0}$ and $L_{0}$ are computed directly from the identification of the calibration marks.

2.3 Object Modeling

The object of interest (fruit) is modeled by a spheroid. Initially, an ellipse is adjusted to the 2D object view, obtained from segmentation, as illustrated in Fig. 1(b, d). This ellipse is used to establish a new coordinate system with X along its major axis and Y along the minor axis. The origin of the XYZ coordinate system is the centre of the spheroid (3D object) and the third axis (Z) is perpendicular to the reference plane.

The 2D ellipse is also used to make an initial estimate for the spheroid minor ($m_{0}$) and major semi-diameters ($M_{0}$), computed at the reference plane level (with $R_{0}$). The spheroid centre height is initially estimated to be $m_{0}$. An iterative process is then performed by computing new estimates for the image resolution at an height $m_{i-1}$ ($R_{i}$), and with this resolution a new value for m ($m_{i}$). The process is repeated until the absolute change in $m_{i}$ is <0.01 mm. The process converges fast, with usually only 4 or 5 iterations required. As an example, for the image presented in Fig. 1(d), the final values were obtained after 5 iterations: $m_{0}=21.62$, $m=m_{5}=19.95$ (the real value of m is 20.0 mm); $M_{0}=32.59$, $M=30.08$ (the real value of M is 30.0 mm).

The intersection of the spheroid with a plane (P) perpendicular to the major axis results in a circle with radius $r_{x}$, which can be computed as a function of the distance between the plane and the centroid (x) using (3). On that circle, the value of z can be computed as a function of x, y by (4), and the object height (H) can be thus estimated as $H(x,y)=z(x,y)+m$.

$$\begin{aligned} \frac{x^2}{M^2}+\frac{r_{x}^2}{m^2}=1 \Leftrightarrow r_{x}=\frac{m}{M}\sqrt{M^{2}-x^{2}} \end{aligned}$$

(3)

$$\begin{aligned} y^2+z^2=r_{x}^2 \Leftrightarrow z=m\sqrt{1-\frac{x^2}{M^2}-\frac{y^2}{m^2}} \end{aligned}$$

(4)

Another important parameter is the spheroid surface area imaged by a pixel. The exact calculation is mathematically and computationally hard, thus a simplified approach is used instead. The surface area (A) for the pixel centered at x, y, z is considered to be approximately the product of two arc lengths: on the ellipse of plane X0Z ($l_{e}$) and on a circle on plane P ($l_{c}$). It is computed by (5–9), where $\Delta$ is the pixel size (in mm) estimated for height H(x, y).

$$\begin{aligned} A=l_{c}.l_{e} \end{aligned}$$

(5)

$$\begin{aligned} l_{c}=r_{x} |\theta _{+}-\theta _{-}| \end{aligned}$$

(6)

$$\begin{aligned} \theta _{\pm }=\cos ^{-1}\left( \frac{y\pm \Delta /2}{r_{x}} \right) \end{aligned}$$

(7)

$$\begin{aligned} l_{e}=\sqrt{\Delta ^{2}+(y_{+}-y_{-})^{2}} \end{aligned}$$

(8)

$$\begin{aligned} y_{\pm }=\frac{m}{M}\sqrt{M^{2}-(x\pm \Delta /2)^{2}} \end{aligned}$$

(9)

Fig. 3.

Auxiliary images modeling the spheroid height (a), resolution (b) and surface area (c), for each pixel (x, y), created for the test image of Fig. 1(c).

Once the spheroid is defined, the x, y coordinates of each pixel in the 2D object are used to compute z and H(x, y) (4), the image resolution R(x, y) (2), $\Delta (x,y)=1/R(x,y)$, and the surface area A(x, y) (5–9). A total of 3 auxiliary images are thus created - H(x, y), R(x, y) and A(x, y), such as the example presented in Fig. 3, produced for the test image of Fig. 1(c). In this example, the positive values of H(x, y) are between 19.97 and 39.90 mm and the Resolution (R(x, y)) between 10.22 and 11.15 pixels/mm, with linear transformations applied to create the grayscale images presented in Fig. 3(a, b). The Surface Area (A(x, y)) of the spheroid pixels varies smoothly, except near the edges where very high values occur. The linear transformation used to produce the greyscale image of Fig. 3(c) only covers values up to 60 ($\times 10^{-3}\,\mathrm{mm}^2$), but the range of values present in this case is from 8.0 to 1609.2 ($\times 10^{-3} \mathrm{mm}^2$).

2.4 Error Estimation

A number of factors influence the accuracy of the measurement of a sub-region area on the object surface. Some are related to the imaging geometry and object shape, which are addressed in the methodology proposed, but others, such as the image resolution and the segmentation process, can also have a substantial contribution. A rough estimate of the error due to these factors is performed in this section, in order to establish what can be a reference baseline.

It is reasonable to consider that the maximum error in the segmentation process along the sub-region contour is 1 pixel (if successful), which will result in a maximum error ($\delta$) of $\pm 2$ pixels/$R_{0}$ in the estimation of the circle diameter. The use of the observed diameter ($d_{obs}$) instead of the reference diameter ($d_{ref}$) will result in an error in the area computed ($A_{obs}$), in regards to the reference area ($A_{ref}$), that can be estimated using (10). An estimate of the relative error ($\varepsilon _{0}$) for the area is obtained by (11). The last term in (10) is considerably smaller that the error term, by a factor of $\delta /2d_{ref}$ (or approximately $1/d_{obs}$, with $d_{obs}$ in pixel units), thus neglected. As an example, the test image presented in Fig. 1(c) has a value of $R_{0}=9.43$ pixels/mm and, considering that $d_{ref}=12.0$ mm, an estimated relative error $\varepsilon _{0}=0.035$ (or 3.5%). In this case the last term in (10) would contribute with only 0.03% to $\varepsilon _{0}$, 2 orders of magnitude smaller than $\varepsilon _{0}$.

$$\begin{aligned} \begin{aligned} A_{obs}&= \frac{\pi }{4} (d_{obs})^{2}&= \frac{\pi }{4} (d_{ref}+\delta )^{2} \simeq A_{ref} + \underbrace{\frac{\pi }{2} d_{ref}\delta }_{error} +\underbrace{\frac{\pi }{4}\delta ^2}_{\approx 0} \end{aligned} \end{aligned}$$

(10)

$$\begin{aligned} \varepsilon _{0} = \frac{|error|}{A_{ref}} = \frac{\frac{\pi }{2}d_{ref}\delta }{\frac{\pi }{4}(d_{ref})^2} = \frac{2\delta }{d_{ref}} = \frac{4}{R_{0}d_{ref}} \end{aligned}$$

(11)

3 Results

The method was evaluated using a wooden prolate spheroid of $40\times 40\times 60$ mm (diameters), and blue circular marks of two sizes: I - $4.0\times$ mm radius ($50.3\times \mathrm{mm}^{2}$ area), II - 6.0 mm radius ($113.1\times \mathrm{mm}^{2}$ area). Two calibration targets were used - of A4 and A5 size. The spheroid volume and surface area [10] are $50265\times \mathrm{mm}^{3}$ and $6767\times \mathrm{mm}^{2}$, respectively.

3.1 Marks on Plane

A preliminary test was carried out, placing the colour marks on the reference plane itself. A total of 20 images were tested - 5 images for each mark (I and II) and calibration target (A4 and A5). The areas computed ($A_{obs}$) were compared with the reference (ideal) values ($A_{ref}$), with the results presented in Table 1. In the worst case scenario (mark I / A4), the average relative error ($\varepsilon _{R}$) is 3.32%, below the maximum expected error ($\varepsilon _{0}$) due to limited image resolution (about 5%). A slightly better case is for mark II (with A4 target), with an average $\varepsilon _{R}$ of 1.15% and $\varepsilon _{0}$ of about 3%. Considering the small size of the marks, it is much better to use an A5 calibration target. The average relative errors with this target are 0.67% (mark I) and 0.72% (mark II), well below $\varepsilon _{0}$ (around 2.5–3.5%).

Table 1. Results for images with marks I and II on the reference plane: area computed ($A_{obs}$) in $\mathrm{mm}^2$, maximum expected error ($\varepsilon _{0}$) and relative error ($\varepsilon _{R}$).

This test provides an estimate of the error that can be expected due to the segmentation process and limited image resolution. This baseline error can thus be expected to be roughly between 2.5% (II/A5) and 5.5% (I/A4) or, more generally, about 4%.

3.2 Marks on 3D Object

Images of the colour mark on the spheroid were acquired in 7 positions, presented schematically in Fig. 4. The figure shows the location of the mark in the 2D view ellipse. In positions #1 to #6 the spheroid is roughly aligned with the calibration target grid, whereas in position #7 the XY axes of the spheroid are at about 45$^\circ$ with the target grid. A total of 28 test images were thus used (7 positions $\times$ 2 marks $\times$ 2 targets). The results are presented in Table 2 for mark I (4 mm radius) and in Table 3 for mark II (6 mm radius). For each image tested, the following values are presented: computed/observed area of the mark ($A_{obs}$) in mm$^2$, relative error ($\varepsilon _{R}$) in %, ellipsoid semi-diameters (m and M) in mm, relative errors ($\varepsilon _{R}$) for the computation of the spheroid surface area and volume.

Fig. 4.

Test position 2D views of the spheroid and mark location. In position $\#7$ the XY axes are aligned at about 45$^\circ$ in regards to the calibration target grid.

The iterative process proved to be effective in establishing a 3D spheroid out of the observed 2D ellipse. The values computed for M and m are generally very close to the real spheroid size ($M=30$ and $m=20$ mm), with average absolute errors of only 0.25 mm for M and 0.20 mm for m (corresponding to relative errors of 0.8% and 1.0%, respectively). For the 28 images tested, the computation of the prolate spheroid volume and surface area [10], using the values M and m obtained from the iterative process, result in average relative errors of 1.77% (for surface area) and 2.71% (for volume).

An initial evaluation of the colour mark area computation is done for the test image of Fig. 1(c) (mark II, A4 target, position #7). The maximum expected error due to the segmentation process and limited image resolution (baseline) was estimated to be about 3.5% (Sect. 2.4). For this image, the computed mark area was 110.18 $\mathrm{mm}^{2}$ (instead of 113.1 $\mathrm{mm}^{2}$), corresponding to a relative error ($\varepsilon _{R}$) of 2.6% (Table 3). It is worth mentioning that if the mark area was computed for the calibration plane, ignoring the fact that the mark is over a 3D object, the result would be 150.4 $\mathrm{mm}^{2}$ ($\varepsilon _{R}$ = 33.0%).

For mark I (Table 2) the relative error in the colour mark area computation is low (up to about 4%) in all 7 positions for the A4 calibration target and 5 positions for the A5 target. For the A5 target, there are 2 observation (positions #2 and #3) where the error is slightly above the expected (5.1% and 6.5%). For mark II (Table 3) the relative error in the colour mark area computation is low (up to about 4%) in 5 out of the 7 positions tested, for both calibration targets. The observation with high relative errors are for positions #4 and #6, for both targets, with relative errors of up to 12.6% (A5 target, position #4).

Table 2. Results for mark I (50.3 $\mathrm{mm}^{2}$) on spheroid - area computed ($A_{obs}$) [$\mathrm{mm}^2$] and relative error ($\epsilon _{R}$) [%], spheroid semi-diameters m and M [mm], relative errors for spheroid surface and volume estimation [%].

Table 3. Results for mark II (113.1 $\mathrm{mm}^{2}$) on spheroid - area computed ($A_{obs}$) [$\mathrm{mm}^2$] and relative error ($\epsilon _{R}$) [%], spheroid semi-diameters m and M [mm], relative errors for spheroid surface and volume estimation [%].

Possible reasons to explain the high relative errors observed in the colour mark area computation observed for some cases, include: (i) the simplification in the computation of the surface area imaged by a pixel; (ii) the fact that the images acquired are not exactly vertical (the principal axis is not perpendicular to the surface observed); (iii) the planar shape of the colour marks. As it happens, the colour marks are plane stickers, thus having a tendency to be slightly detached from the spheroid, particularly in areas of strong curvature, as those close to the edge of the spheroid along its major axis. Furthermore, the spheroid surface is not perfect, thus limiting the ability of the sticker to properly adhere to the surface. But the most likely reason is (i), as the arc length along the ellipse of plane X0Z tends to be underestimated. It is a matter for further investigation, preferably with an experimental evaluation with spheroids of various sizes.

Overall, the results are in line with the expected errors (baseline), except for 6 out of 28 images tested. The relative error in the computation of the mark area of those 6 cases is nevertheless acceptable. In fact, even in the worst cases (up to 12.6% error) the results are much better than what would be obtained if the 3D shape of the object was ignored (which would result in relative errors up to 80% for the images tested).

The errors for position $\#1$ and $\#7$ are comparable, which indicates that the orientation of the spheroid XY axes in regard to the target grid is not critical. Generally it is preferable to have the colour mark as close to the centre of the imaged ellipse as possible (such as positions #1 and #7). Given the rather small size of the colour marks tested, the results could be improved by using a higher resolution camera. It is worth mentioning that the 5MP camera used can be considered to be close to the bottom level of current devices (smartphones and tablets).

4 Conclusion

The method proposed in this paper can be useful for the computation of sub-region surface area of a 3D object, modeled by a spheroid, using a single image acquired with a standard camera and a simple calibration target. The experimental evaluation indicates that the proposed iterative process is effective in establishing a 3D spheroid out of the observed 2D ellipse, with average relative errors of 0.8% and 1.0% in the estimation its semi-diameters, which result in low errors in the computation of both the spheroid surface area (1.77%) and volume (2.71%). The computation of the colour mark surface area proved to be effective regardless of the mark location in the 2D ellipse, with errors up to about 4% in 22 out of 28 images tested.

The application of the proposed methodology to fruits lesions poses potentially new challenges. A major one is the segmentation, both of the fruit and the lesion, which is a critical issue for the subsequent processing. Of particular concern is the lesion, which might be difficult to properly distinguish. Other issues include the fruit shape (it might not be similar to a spheroid), fruit surface roughness, and the actual image acquisition geometry, which can be difficult to control in field. There is nevertheless a great potential in having low cost, easy to use systems to acquire and process in field data for fruit lesion area estimation. The methods presented in this paper provide a set of tools that can be a basis for such system.