Detection of dropped citrus fruit on the ground and evaluation of decay stages in varying illumination conditions

doi:10.1016/j.compag.2016.05.020

Computers and Electronics in Agriculture

Volume 127, September 2016, Pages 109-119

https://doi.org/10.1016/j.compag.2016.05.020 Get rights and content

Highlights

•
An image brightness enhancement algorithm for outdoor images was proposed.
•
Images had constant brightness after enhancement regardless of the original brightness.
•
Applied canny edge density was proposed to choose the best citrus-containing circle.
•
The detection algorithm showed different performances depending on size of sub-image.
•
Highest performance was 89.6% for correct identification and 5.0% for false positive.

Abstract

The devastating disease Huanglongbing (HLB) has greatly affected citrus in Florida and other growing regions. Detecting dropped fruit is one method of estimating the presence and severity of the disease. The purpose of this study was to develop a machine vision system that can detect dropped citrus on the ground in varying illumination conditions and identify decaying stages of the dropped fruit. In this paper, a novel method for image brightness correction using a contrast limited adaptive histogram equalization was developed to produce constant image brightness levels between and within images. Objectives of this study were to: (1) solve the varying illumination problem and create a consistent brightness level between and within the images, (2) develop an algorithm to eliminate multiple detections of a single fruit from the circular Hough transform, (3) design an algorithm to evaluate decaying stages of the dropped citrus, and (4) demonstrate ability to create a fruit drop map of citrus at each decaying stage in a commercial citrus grove. The result shows all processed images had desired brightness levels (152 out of 255) with a standard deviation of 1.0. Correct identification of fruit and false positives were measured as 89.6% and 5.0%, respectively. False classifications of decay stages of fruit were as low as 4.2% and 18.5% for recently dropped fruit and rotten fruit, respectively. The techniques developed in this work could be further developed into a commercial machine vision system for a real-time dropped fruit mapping system.

Introduction

Citrus Huanglongbing (HLB) or citrus greening has become a devastating disease in the United States, Brazil, and other citrus-producing regions. Citrus production in the United States was estimated to 5.8 million tons in 2014–2015 crop year which was 350,000 tons lower than previous season, mainly due to the decreased production in Florida (USDA, 2015a). The widespread cases of the disease in Florida has caused production to decrease 34.4% from 189.1 million boxes in the 2008–2009 crop year to 124.0 million boxes in the 2013–2014 crop year (USDA, 2010, USDA, 2015b). There is no known cure for the disease, but it is important to know where the disease is located and the severity at those places so that mitigating actions can be taken. One approximate indicator of severity is the number of dropped fruit. However, manual inspection to find the amount of the dropped fruit increases the production costs due to labor cost (Choi et al., 2015). Using colors and shapes of citrus fruit, and the machine vision technique, the manual inspection of the dropped citrus can be automated.

However, there are common difficulties in detecting citrus fruit using outdoor images. First of all, developing an outdoor machine vision system is challenging due to varying illumination conditions among images. Many studies reported that the varying illumination caused reduced accuracy of machine vision systems for outdoor fruit detection applications (Annamalai and Lee, 2003, Annamalai et al., 2004, Stajnko et al., 2004, Patel et al., 2012, Wachs et al., 2010, Kurtulmus et al., 2011, Wang et al., 2012).

Annamalai and Lee (2003) developed a machine vision system for citrus yield mapping. However, the varying brightness levels in their images caused incorrectly detected background and missed fruit. Even though their algorithm included several processes to reduce the false positives, the system showed a performance of only R² = 0.76 between the actual number of fruit and estimated data. Wachs et al. (2010) also stated in their article that uncontrolled natural illumination was one of the reasons for limited performance. They developed a machine vision system using color and geometric properties to detect apples in orchards. However, the color was severely affected by unconstrained illumination. Although they combined thermal images to improve varying illumination conditions, their algorithm yielded a high false positive rate (46.8%) because their algorithm was based on the assumption that all images had the same illumination level.

Various methods were developed in many studies to overcome varying image illumination. Most of methods were based on two approaches. The first approach is to control the hardware system during image acquisition. Annamalai et al. (2004) used different shutter speeds of a camera for each image in a real-time citrus yield estimation system. They manually adjusted the shutter speed according to the light conditions. However, uneven illumination within an image was not solved by adjusting the shutter speed, and it reduced the performance during segmentation using colors. Wang et al. (2012) developed a system to evaluate the quality of cherries using their skin color. They used a flashlight with small camera aperture in order to decrease the effect of varying illumination. However, they reported that the flashlight caused highly saturated areas on fruit surfaces, and they had to use different color models for each aperture size.

The second approach to solve the varying illumination is to use software for pre-processing operations on the images. Chinchuluun and Lee (2006) adopted a gamma correction algorithm to decrease varying illumination conditions in their machine vision system for citrus yield mapping. It showed better accuracy by applying the pre-processing steps, achieving R² = 0.83 between the actual number of fruit and fruit count by the algorithm. Kurtulmus et al. (2011) developed a green citrus detection system using histogram equalization and a logarithmic transform to enhance the image brightness. Even though the image brightness was corrected, the classification algorithm showed different results depending on brightness levels of the original images. For dark images, it had a 75.2% correct identification rate and a 16.8% of false positive rate. But for the bright images, the correct identification rate was 75.4%, and the false positive rate was 40.3%. Choi et al. (2013) proposed an illumination normalization technique to develop a fruit drop detection system. The normalization was processed by dividing each RGB color component by the average gray value of all the pixels in the image. The result showed an 81.3% correct identification rate with a 12.9% false positive rate. The results from the mentioned studies show that image enhancement algorithms for preprocessing yielded better performances compared to the method controlling the hardware. The gamma correction, the logarithmic transform, and the histogram equalization are well-known methods for image enhancement. However, the main purpose of the three methods is to enhance the quality of an image. Therefore, there is no guarantee that the algorithm would reduce the differences in illumination between images. In fact, none of the above studies showed analysis about improving the consistency of image brightness between images. Therefore, in this study a pre-processing step using a contrast limited adaptive histogram equalization (CLAHE) (Zuiderveld, 1994) was introduced to have a constant image brightness level not only within an image but also between images.

The second challenge is that to keep the same distance from a camera to the fruit is difficult in an outdoor machine vision system resulting in a wide range of fruit sizes among the images. A circular Hough transform (CHT) is the commonly used method to find potential fruit areas using shape information. However, due to the wide range of the fruit sizes in the images, the CHT can cause multiple detections of a single fruit with different sizes of circles (Bansal et al., 2013, Sengupta and Lee, 2014, Silwal et al., 2014). To locate accurate fruit areas without multiple detections, Bansal et al. (2013) suggested using an average of centers of overlapped fruit. Another approach was proposed by Sengupta and Lee (2014) to select the largest radius circle among the overlapped ones. However, averaging the circles’ locations or picking the largest circle disregarded the actual context within the detected areas and caused the choosing of an inappropriate circle containing not only the fruit region but also some of the background. Therefore, in this study, a new technique, applied canny edge density (ACED) algorithm, was developed with a canny edge detector and a morphological closing operation to locate the most proper fruit area considering the context of the detected circles from the CHT.

The overall goal of this study was to develop a machine vision system to estimate the amount of fruit drop in a commercial citrus grove. Specific objectives were:

1.
To develop an algorithm to solve varying illumination condition and have constant image brightness throughout all acquired images.
2.
To develop an algorithm to remove multiple detections of a single fruit from the circular Hough transform.
3.
To develop an algorithm to evaluate decaying stages of the citrus so that citrus growers can understand when fruit dropping happened.
4.
To create a fruit drop map of citrus at each decaying stage.

Section snippets

Image acquisition hardware

The image acquisition hardware was developed to (1) acquire images and GPS coordinates, (2) decrease the impact of varying illumination, and (3) protect the cameras. To achieve a continuous image acquisition process, the whole system was developed to be mounted to an all-terrain vehicle (Fig. 1a, Sportsman, Polaris Industries Inc., Medina, Minnesota). Two CMOS cameras (Sony ActionCam, Tokyo, Japan) were installed under a metal shield and recorded videos while the vehicle was driving in a citrus

Result of image enhancement using CLAHE

Fig. 12 shows a comparison of original images (Fig. 12a and b), enhanced images using the standard histogram equalization (Fig. 12c and d), and enhanced images using the CLAHE (Fig. 12e and f). Average brightness levels of the original images were in different ranges, 75.8 and 100.3. After the image enhancement using the HE, the brightness levels became similar, about 127 for both images. However, the center of the images was excessively contrasted, changing many pixels to a darker color. The

Conclusion

A machine vision system was developed for counting dropped citrus fruit under tree canopies and evaluating its stages: recently dropped fruit or rotten fruit. The contrast limited adaptive histogram equalization (CLAHE) enhanced the brightness of all images to have the constant image brightness levels. The proposed algorithm improved accuracy of the classification compared to previous studies. The performance of fruit detection algorithm was the highest (89.6% for correct identification and

Acknowledgement

This project was supported by the University of Florida/Institute of Food and Agricultural Sciences’ Citrus Initiative program. The authors would like to thank Mr. Michael Zingaro, Dr. Alireza Pourreza, Dr. Mazin Saber, Ms. Mubarkat Shuaibu and Ms. Brianna Posadas at the University of Florida for their assistance with this research.

References (24)

H.D. Cheng et al.
Approaches for automated detection and classification of masses in mammograms
Pattern Recogn.
(2006)
F. Kurtulmus et al.
Green citrus detection using ‘eigenfruit’, color and circular Gabor texture features under natural outdoor conditions
Comput. Electron. Agric.
(2011)
S. Sengupta et al.
Identification and determination of the number of immature green citrus fruit in a canopy under different ambient light conditions
Biosyst. Eng.
(2014)
D. Stajnko et al.
Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging
Comput. Electron. Agric.
(2004)
Q. Wang et al.
Outdoor color rating of sweet cherries using computer vision
Comput. Electron. Agric.
(2012)
Annamalai, P., Lee, W.S., 2003. Citrus yield mapping system using machine vision. In: Paper Presented at the 2003 ASAE...
Annamalai, P., Lee, W.S., Burks, T.F., 2004. Color vision system for estimating citrus yield in real-time. In: Paper...
R. Bansal et al.
Green citrus detection using fast Fourier transform (FFT) leakage
Precis. Agric.
(2013)
L. Breiman
Random forests
Mach. Learn.
(2001)
Chinchuluun, R., Lee, W.S., 2006. Citrus yield mapping system in natural outdoor scenes using the watershed transform....

Choi, D., Lee, W., Ehsani, R., 2013. Detecting and counting citrus fruit on the ground using machine vision. In: Paper...

D. Choi et al.

A machine vision system for quantification of citrus fruit dropped on the ground under the canopy

Trans. ASABE

(2015)

Cited by (17)

Transforming unmanned pineapple picking with spatio-temporal convolutional neural networks
2023, Computers and Electronics in Agriculture
Automated pineapple harvesting has emerged as a prominent prospective development within the agricultural domain. Nevertheless, the intricate growth conditions that pineapples encounter in the field, such as inadequate light, overexposure, obstructions caused by fruit leaves, or the overlapping of fruits, pose substantial challenges to the accuracy and robustness of traditional real-time detection algorithms. In recent times, Transformer models, when applied to computer vision, have exhibited commendable performance, underscoring their potential for target detection in smart agricultural applications. In this report, we propose a spatio-temporal convolutional neural network model that leverages the shifted window Transformer fusion region convolutional neural network model for the purpose of detecting pineapple fruits. Our study includes a comparative analysis of these results and those obtained through the utilization of conventional models. Additionally, we investigate the influence of various aspects of data preparation, including image resolution, object size, and object complexity, on the ultimate pineapple detection outcomes. Experimental findings elucidate that, in the case of detecting a single-category target like a pineapple, the employment of 2000 annotated supervised data points yields the optimal detection accuracy. Further augmenting the size of the training dataset does not yield any significant improvement in detection accuracy. Furthermore, images of pineapples captured from greater distances encompass smaller targets and an increased number of pineapple instances, rendering them more intricate and challenging to accurately detect. In summary, our study employs the spatio-temporal convolutional neural network model to attain pineapple detection with an impressive accuracy rate of 92.54% and an average inference time of 0.163 s, thus affirming the efficacy of our developed model in achieving superior detection results.
Development of a Handheld IoT-Based Fruit Harvester to support Agrotourism
2022, Microprocessors and Microsystems
Citation Excerpt :
Several efforts can be made to reduce the risk of damage, including the application of precision agriculture. Precision agriculture, a method of managing agricultural variables to increase profits in the farming industry [2–4], utilizes advanced technologies, such as positioning system technology [5], to obtain real-time mapping [6–8], accurate mapping of crops, and reduce costs. Later, the GPS data acquired will be supported by variables, such as land processing costs, labor, and harvest yields.
Agrotourism, a development in the tourism industry that can increase the economic potential of an area, offers many activities, such as fruit picking tours. Fruit picking, a crucial part of the harvesting process, involves separating fruits from plants to obtain economic value. However, several common mistakes during harvesting can damage the plants or branches. In this study, we developed a mechanical fruit harvester incorporated with the Internet of Things (IoT). The harvester has multiple functions, such as minimizing the damage to tree branches, monitoring the fruit productivity of each plant, and providing information on fruit weight and prices in real-time. The tool is equipped with a smartphone, several infrared and load sensors, a microcontroller, a cloud data logger, and a data analysis system. We tested the accuracy of the GPS on several smartphones while in agrotourism areas. The accuracy of the GPS on the harvesting tool is approximately 3 meters, and the success rate in calculating the number of fruits using the infrared sensor is 93.3 %, depending on the diameter of the fruit passing into the bucket. In addition, the harvester is equipped with a load sensor with a measurement deviation rate of 2.48%.
Overcurrent-driven LEDs for consistent image colour and brightness in agricultural machine vision applications
2021, Computers and Electronics in Agriculture
Citation Excerpt :
Gongal et al. (2016) used a full canopy cover made with tarpaulin to image individual trees. Choi et al. (2016) used a metal shield housing to detect dropped fruit under tree canopy. Even though LEDs are less powerful than xenon lamps, they are more efficient in terms of lumen output per wattage.
Machine vision systems are being utilized extensively in agriculture applications. Daytime imaging in outdoor field conditions presents challenges such as variable lighting and colour inconsistencies due to sunlight. Motion blur can occur due to vehicle movement and vibrations from ground terrain. A camera system with active lighting can be a solution to overcome these challenges. In this study, the usage of over-current driven LEDs to produce a powerful flash was investigated as a viable light source for daytime imaging. The current drawn by an LED was increased by a factor of six times its normal rating resulting in increased illuminance. A circuit was designed for storing and releasing energy to the LEDs for a strobe-like effect and a controller was used for synchronizing the strobe with a camera to acquire images. The system was deployed in an apple orchard on three days in summer of 2020. Images were taken throughout the day in both sunny and cloudy conditions of different canopy structures. There was substantial improvement in image brightness and colour consistency by using the LED flashes. Images captured by the prototype system during an 11-hour period showed an average decrease of 85% in standard deviation for the Hue-Saturation-Value (HSV) channels compared to that of the auto-exposure setting. Additionally, the prototype system was able to fix motion blur in images averaging 7 mm in error for a stereo vision application with the camera moving at 7 km/hr. These results show that the designed LED flash system can reduce the undesirable effects of lighting variability and motion blur in images stemming from outdoor field conditions.
A role of computer vision in fruits and vegetables among various horticulture products of agriculture fields: A survey
2020, Information Processing in Agriculture
Citation Excerpt :
ISADH feature extraction produces a higher accuracy rate (>93%) in both RGB and HSV color space and UNSER has the lowest accuracy rate among all feature extraction. Mean and standard variance performance metric are also used by the author [91]. Mean and standard variance are applying to get a performance by the author [92] They have also compared the performance of the proposed system with another metric such as sensitivity, specification, accuracy, mean and last Square error.
Computer vision is a consistent and advanced technique for image processing, with the propitious outcome, and enormous potential. A computer vision has been strongly adopted in the heterogeneous domain including agriculture. During the study of existing research on the role of computer vision in fruits and vegetables among various horticulture products of agriculture fields it is noticed that, the existing survey paper has not focused properly on mathematical framework, feature descriptor, defect detection on multiple datasets of fruits and vegetables elaborately. This has motivated us to undertake an extensive survey. In this paper, we examine the paper broadly related to fruits and vegetables among various horticulture products of agriculture fields, specific model, data pre-processing, data analysis method and overall value of performance accuracy by using a particular performance metric. Moreover, we study the different type of disease present in various fruit and vegetable. We have also focused on the comparison of different machine learning approach with respect to different performance metrics on the same dataset. Thus, we have found that among all existing machine learning techniques SVM give better classification accuracy. A generalized framework to grade the quality and defect detection of multiple fruits and vegetables is also proposed in this survey. This paper covers the survey of ninety-eight papers closely related to computer vision in the agricultural field. By the survey, we have found that computer vision plays an important role and has a large potential to address the challenges related to the agricultural fields.
Evaluation of Citrus Gummosis disease dynamics and predictions with weather and inversion based leaf optical model
2018, Computers and Electronics in Agriculture
Citation Excerpt :
However, the population greater than 10 propagules/cc in soil significantly affects the fibrous root density when treated with fungicides (Naqvi and Singh, 2002). By understanding the economic value of Citrus, many research work has also initiated on citrus fruit and leaves diseases, use of technology in citrus farming, use of spectral analysis for feature identification in citrus disease or production management (Carrer et al., 2017; Choi et al., 2016; Maldonado and Barbosa, 2016; Sharif et al., 2018; Diaz et al., 2017) There is another case about the effect of lack of transpiration and photosynthesis activity due to the presence of pathogen on plant physiology (Fig. 1).
One of the major threats for crops around the world due to pest and diseases, which can impact the health, economy, environment, and society at large. In general, several issues related to crop yield improvement arises due to insufficient and inadequate knowledge. Therefore, there is a need to develop viable models that incorporate various weather-soil-plant factors, which can give better understanding of the crop and enable timely interventions for yield improvement. To overcome Citrus Gummosis disease related issues and increase the Citrus productivity, seven different datasets Temperature (T), Humidity (R_h), Rainfall (R), Soil Moisture (SM), Soil Temperature (ST), Leaf Area Index (LAI) and Chlorophyll (C_ab) were used. Considering various plant, soil and environmental factors, the Citrus Gummosis prediction model has been developed with the multi-source datasets from June 2014 to November 2016 using Support vector regression (SVR) and multilinear regression (MLR). The research is carried out for healthy (5–10 Yrs. and 11–15 Yrs.) and unhealthy (5–10 Yrs. and 11–15 Yrs.) age group of plants. Inverse PROSAIL model has been simulated for retrieving citrus C_ab and LAI values. These values were validated with the actual field data. Both the weather and soils based disease prediction models has been developed and validated with MLR and SVR. Further, the influence of Gummosis disease on plant parameters was also studies with the new contribution of Biophysical variables (LAI and C_ab) based statistical prediction model. The SVR model gave fairly good performance as compared to MLR. In addition to the separate models a the combined scenario approach (Integrated Gummosis Disease Forecast Model: IGDFM) is designed to understand the interconnectivity of the parametric conditions (weather-soil- plant parameters) with disease physiology with respect to different age group of the plants. The RMSE of proposed approach for higher age group plants (i.e. 11–15 years) in the combined scenario was 0.9061 and 0.8518 for SVR and MLR methods, respectively. It is envisaged that this study could enable farmers to recognize and predict the timing and severity of the Gummosis disease in Citrus and thereby achieve yield improvement.
Release and recovery of pectic hydrocolloids and phenolics from culled citrus fruits
2017, Food Hydrocolloids
Citation Excerpt :
Huanglongbing (HLB) is a bacterial disease that has spread throughout the citrus growing regions of Florida, and much of the citrus cultivating regions in the world (Choi, Lee, Ehsani, Schueller, & Roka, 2016; Gottwald, 2010).
The citrus industry worldwide is threatened by a bacterial disease (Huanglongbing, HLB) spread by a sap sucking hemipteran, the Asian citrus psyllid (Diaphorina citri). Before tree death there is a period of increased preharvest fruit drop and the production of smaller fruit containing off-flavored juice. The increasing frequency of HLB symptomatic fruit moving into the juice processing chain may become a challenge for maintaining flavor quality. An alternative use is sought for flavor degrading fruits, to recover value for fruit growers and juice processors. The purpose of this study was to investigate a steam explosion process for the release and recovery of highly functional pectic hydrocolloids and phenolic compounds from culled fruit. Symptomatic fruit from two varieties were culled and either entire fruits or juice-extracted fruit peels were submitted to steam explosion. Released pectic hydrocolloids, a useful hydrocolloid readily functionalized via alkaline demethylesterification, were co-isolated with a solvent-free fraction enriched in phenolic compounds. Recovery of pectic hydrocolloids ranged from 58% to 78%. Weight average molecular weight of this material ranged from 5.78 × 10⁵ to 1.02 × 10⁶ kDa and the degree of methylesterification ranged from 61% to 79%. Functionalization using alkaline, calcium sequestering compounds resulted in viscosities up to 250 mPa s⁻¹. Additionally, recovery ranges of 5%–100% of the polymethoxylated flavones, 2%–71% of the flavanone glycosides, >50% of the limonoids were obtained. These results indicate this process would be useful for recovering value from culled fruit which would otherwise either degrade juice quality or be shunted for conversion to low value animal feed.

View all citing articles on Scopus

View full text

Original papersDetection of dropped citrus fruit on the ground and evaluation of decay stages in varying illumination conditions

Highlights

Abstract

Introduction

Section snippets

Image acquisition hardware

Result of image enhancement using CLAHE

Conclusion

Acknowledgement

Pattern Recogn.

Comput. Electron. Agric.

Biosyst. Eng.

Comput. Electron. Agric.

Comput. Electron. Agric.

Green citrus detection using fast Fourier transform (FFT) leakage

Precis. Agric.

Random forests

Mach. Learn.

A machine vision system for quantification of citrus fruit dropped on the ground under the canopy

Trans. ASABE

Original papers
Detection of dropped citrus fruit on the ground and evaluation of decay stages in varying illumination conditions