A robust algorithm based on color features for grape cluster segmentation

doi:10.1016/j.compag.2017.08.025

Computers and Electronics in Agriculture

Volume 142, Part A, November 2017, Pages 41-49

https://doi.org/10.1016/j.compag.2017.08.025 Get rights and content

Highlights

•
A robust algorithm based on ANN and GA for segmenting grape clusters and leaves from background using color features was developed.
•
GA was employed for optimizing of ANN structure and selecting supreme color features simultaneously.
•
The overall accuracy of the pixel classification of the algorithm was 99.40%.

Abstract

Image processing has been widely used for automation purposes in modern agriculture. The algorithm development for the image segmentation is the most controversial and challenging issue in orchard environment which researchers encounter. This paper describes a robust algorithm based on artificial neural network (ANN) and genetic algorithm (GA) for segmenting grape clusters from leaves and background using color features near to harvest. GA was employed for optimizing of ANN structure and selecting supreme color features simultaneously. The results showed that GA specifies the 8 color features as supreme features and define 8–15-35–3 as the best structure of the ANN. The overall accuracy of the algorithm was 99.40%. The promising results in algorithm development described in this study lead to introduce it as a practical sensing tool in precision agriculture as well as those industrial facilities dealing with image analysis.

Introduction

There are several studies in the literature where image analysis has been applied to detect agricultural products. Mora et al. (2016) proposed an image segmentation algorithm to exclude contributions of non-leaf materials from leaf material in the analysis. Shapiro et al. (2009) and Steward et al. (2002) developed an image processing algorithm to detect the date palm and weed targets, respectively and obtained results were used for further operations. Amatya et al. (2015) investigated the image processing algorithm to branch detection for cherry harvesting. They used Bayesian classifier and R, G and B color features of each pixel for segmenting branch pixels and achieved 89.6% branch pixel classification accuracy.

Color features have been used for red and green apple recognition by Zhou et al. (2012). Their results showed that color difference features (R-G, R-B, G-R, and R-G) had good potential for segmentation of apple fruits form background (leaves, branches, and sky). Another study was carried out on apple identification based on color and shape features in order to estimate crop-load (Gongal et al., 2016). Grayscale images were obtained by subtracting green and blue channels from red channel (R-G and R-B). Then Otsu’s threshold method was used to convert the gray images to binary images. Canny edge detector was carried out to detect edge and Circular Hough Transformation (CHT) was applied to identify circular objects that are likely to be apples. The apple identification accuracy was 79.8% in their study.

Diago et al. (2012) were developed an approach by generating a supervised classifier based on the Mahalanobis distance to characterize the grapevine canopy, and assessed leaf area and yield using RGB images. The segmentation algorithm was able to distinguish the leaves with 92% and grape cluster with 98% accuracy. The ability to segment was depended on the contrast between black grapes with leaves and background. Berenstein et al. (2010) developed an image processing algorithm for detection of grape cluster based on Pearson's correlation method for deleting the effective features and a decision tree algorithm for classification. Their obtains showed that mean gradient magnitude of the R, G, B and V channels were supreme features for segmenting the grape cluster area from foliage area. Also, they found that grape cluster detection leads to 30% reduction in use of pesticides. Nuske et al. (2014) presented the image processing technique for yield estimation in the vineyard. Their results demonstrated that color features and texture-descriptors worked very well in different light conditions for grape cluster segmentation. In addition, results indicated that total yield could be estimated with an average error between 3% and 11%.

Artificial neural network (ANN) and genetic algorithm (GA) and their combination have shown to be powerful tools for solving the complex problem that has not exact solutions especially in image segmentation area (Al-Mohair et al., 2015, Khan and Jaffar, 2015).

Important steps in improving the performance of the ANN classifier are to find the best structure of the ANN and select the best features used as input to the ANN for classification. The ANN structure has two major parameters: number of hidden layers and neurons in each hidden layers and one of the common methods used by authors for finding the best structure is train and error method (Salehi et al., 2011, Behroozi-Khazaei et al., 2013). Selecting the best features for classification means that those features that do not help to improve the classification accuracy should be removed from the feature vector. Nevertheless, an effective feature vector must contain sufficient elements for classification (Omid et al., 2010, Banakar et al., 2017). Appling the soft computing method such as GA for finding the best structure of the ANN and best features selection is interesting and growing.

The GA was already used for optimizing the ANN structure (Mohebbi et al., 2011) and for feature selection (Vafai and De Jong, 1992, Chtioui et al., 1998). The feed-forward neural networks are also the most popular architectures due to flexibility architecture and good representational capabilities (Salehi et al., 2011). Liu et al. (2016) performed the ANN technique for on-tree recognition of apple at nighttime for apple harvesting robots. The ANN has adopted a classical topological structure with 3 layers and the number of nodes in hidden layer was determined by an empirical formula. The color components of RGB and HSI are used as input data for ANN. Also for better segmenting of regions around the edge of apples, the color difference between recognized regions and edge (R1–R2, G1–G2, B1–B2) and the distance of them are extracted which is applied as input data of the second ANN. Their results indicated that proposed algorithm worked very well for apple segmenting with 97% accuracy. Furthermore, Zheng (2005) developed an algorithm for tree crown detection based on ANNs. The tree crown features were extracted after image segmentation. These features were put into the designed ANN, and six unknown typical tree crowns (cone, oval, spherical, cylindrical, umbellate, and truncate) recognized precisely and effectively. No quantitative results were found. Cheng et al. (2017) were developed the new approach of using image analysis and tree canopy features to predict the first reliable time of yield (early yield) with ANN. For each canopy image, pixels were segmented into fruit, foliage, and background using image segmentation. The four features extracted from the image data set: total cross-sectional area of fruits, fruit number, total cross-section area of small fruits, and cross-sectional area of foliage that were used as inputs and actual weighted yield per tree was used as output. The results showed that correlation coefficients (R2) between the estimated and the actual weighted yield, mean forecast error (MFE) measures the unbiasedness between the estimated and the actual weighted yield, mean absolute percentage error (MAPE), and root mean square error (RMSE) were 0.81, −0.05, 10.7%, 2.34 kg/tree, respectively.

As mentioned earlier, image processing has been used to assist with precision agriculture practices specially yield estimation. On the other hand, Gongal et al. (2015) were reported that one of the issues which can increase the overall accuracy of fruit detection is the improvement of image segmentation method. While the fruit segmentation is the issue when the green/greenish fruits are surrounded by green leaves, a color segmentation method is not an easy task. Therefore, a robust algorithm is needed for fruit detection with high accuracy regardless of fruit color and application environment. The objective of this study was to present a robust method based on ANN and GA for grape cluster segmentation when the grape is getting mature. The core idea was to determine the number of hidden layers and number of neurons in hidden layers of the ANN and selecting the best color features as input in the ANN which was simultaneously performed with a genetic algorithm technique.

Section snippets

Materials and methods

Developing the proposed algorithm involves three phases; first phase dealing with developing the image processing algorithm for extracting the color features and preparing the dataset matrix, second phase dealing with selecting the supreme color features from the dataset and best structure of the ANN as classifying algorithm, and the third phase dealing with evaluating the performance of the developed algorithm. For developing this algorithm MATLAB software was used. The following sections

Results and discussion

Fig. 6 illustrates the variation of the best fitness value with generations. As shown, the algorithm automatically searches for the appropriate chromosome according to the given objective. Therefore, the value of best fitness is decreased across generations and the best fitness value emerged at the 5th generation and after that its value has not any change until 30th generation. For this reason, the optimization process stopped at 30th generation. The optimization process resulted in selection

Conclusions and future works

The image processing techniques have been used across a vast range of agricultural productions and many have been trying to develop a stronger algorithm for approaching perfection. In the current study, the color features were considered as the basis for image segmentation in vineyard. The core concept was to use the genetic algorithm and artificial neural network structure for selecting the supreme color features. The novel proposed algorithm was limited to detect the grape cluster located

References (35)

H.K. Al-Mohair et al.
Hybrid human skin detection using neural network and K-means clustering technique
Appl. Soft Comput.
(2015)
N. Behroozi-Khazaei et al.
Applied machine vision and artificial neural network for modeling and controlling of the grape drying process
Comput. Electron. Agri.
(2013)
A. Gongal et al.
Sensors and systems for fruit detection and localization: a review
Comput. Electron. Agric.
(2015)
A. Gongal et al.
Apple crop-load estimation with over-the-row machine vision system
Comput. Electron. Agric.
(2016)
A. Khan et al.
Genetic algorithm and self-organizing map based fuzzy hybrid intelligent method for color image segmentation
Appl. Soft Comput.
(2015)
X. Liu et al.
A method of segmenting apples at night based on color and position information
Comput. Electron. Agric.
(2016)
K. Mollazade et al.
Comparing data mining classifiers for grading raisins based on visual features
Comput. Electron. Agric.
(2012)
M. Mora et al.
Automated computation of leaf area index from fruit trees using improved image processing algorithms applied to canopy cover digital photography
Comput. Electron. Agric.
(2016)
M. Mohebbi et al.
Prediction if moisture content in pre-osmosed and ultrasound dried banana using genetic algorithm and neural network
Food Bioprod. Process.
(2011)
C. Onnen et al.
Genetic algorithms for optimization in predictive control
Control Eng. Pract.
(1997)

M. Omid et al.

Development of pistachio sorting system using principal component analysis (PCA) assisted artificial neural network (ANN) of impact acoustics

Expert Syst. Appl.

(2010)

S. Amatya et al.

Detection of cherry tree branches with full foliage in planar architecture for automated sweet-cherry harvesting

Biosys. Eng.

(2015)

A. Banakar et al.

Combined application of decision tree and fuzzy logic techniques for intelligent grading of dried figs

J. Food Process Eng.

(2017)

R. Berenstein et al.

Grape clusters and foliage detection algorithms for autonomous selective vineyard sprayer

J. Intell. Service Robot.

(2010)

H. Cheng et al.

Early yield prediction using image analysis of apple fruit and tree canopy features with neural networks

J. Imag.

(2017)

M. Diago et al.

Grapevine yield and leaf area estimation using supervised classification methodology on RGB images taken under field conditions

Sensors

(2012)

G. Dunn et al.

Yield prediction from digital image analysis: a technique with potential for vineyard assessments prior to harvest

Aust. J. Grape Wine Res.

(2004)

Cited by (50)

Improving crop image recognition performance using pseudolabels
2024, Information Processing in Agriculture
In crop image recognition, when faced with a large quantity of unlabeled data, the traditional manual labeling method requires a large amount of human and material resources. To solve this problem, this study proposes an image recognition method based on a pseudolabeling technique. First, the data are divided into labeled and unlabeled data. The initial network model is trained on labeled data. Then, pseudolabeling of the unlabeled data is predicted, and only the data that satisfy the confidence threshold are regarded as valid pseudolabeling. To convert the unlabeled data into supervised training data, the two types of data are mixed. The training is terminated when the number of remaining unlabeled data satisfies the end condition and when the fivefold cross-validation method is used to evaluate model performance. Compared with the traditional semisupervised method, the experimental method is simpler and more applicable. Experiments were conducted on rice growth stage recognition and crop weed seedling recognition tasks. The results showed that the proposed method achieved 99.17% accuracy in rice growth stage recognition and a high AUC value of 99.93% in crop weed seedling recognition, which demonstrated excellent performance. Compared with the traditional model, this method not only improves in accuracy but also has better stability and wider applicability and is expected to provide an efficient, accurate and scalable solution for crop image recognition.
Filament-necking localization method via combining improved PSO with rotated rectangle algorithm for safflower-picking robots
2023, Computers and Electronics in Agriculture
Safflower is one of the most important specialty economic crops in the world. Safflower blossoms are harvested continuously for 3–5 times. Untimely picking can affect the opening of filaments in the next crop, resulting in reduced filament production and serious economic losses. However, the safflower images collected by picking robots were affected by lighting conditions or complex backgrounds. The existing segmentation algorithms have the problem of insufficient segmentation or over-segmentation, because of the small safflower size and localization area. Therefore, combining improved particle swarm optimization (PSO) with a rotated rectangle algorithm based on the filament-necking localization method for safflower-picking robots was proposed. The R-component image in RGB color space was extracted as a preprocessing sample by combining the color features of safflower. The inertia weights in the PSO algorithm are improved to enhance the performance of the algorithm. An adaptive nonlinear function is introduced to search for the optimal threshold and initially segmented to obtain the binary image. Then, the barycenter and minimum outer rectangle of the contour were set up using the rotated rectangle algorithm based on the geometric features of the filaments. The circular region of interest (Cir-ROI) of filament-necking is determined. The Zhang-Suen refinement algorithm was used for skeleton extraction to design an algorithm for localizing the picking point of safflower filaments. To test safflower images collected in complex environments, the results of average processing time were 0.14 s, and the average relative target area error rate was 19.33 %. Moreover, the localization accuracy of the picking point was 89.75 %. The filament-necking localization method provides a theoretical basis and experimental data support for damage reduction and efficient harvesting of safflower filaments.
Application of AI techniques and robotics in agriculture: A review
2023, Artificial Intelligence in the Life Sciences
The aim of the proposed work is to review the various AI techniques (fuzzy logic (FL), artificial neural network (ANN), genetic algorithm (GA), particle swarm optimization (PSO), artificial potential field (APF), simulated annealing (SA), ant colony optimization (ACO), artificial bee colony algorithm (ABC), harmony search algorithm (HS), bat algorithm (BA), cell decomposition (CD) and firefly algorithm (FA)) in agriculture, focusing on expert systems, robots developed for agriculture, sensors technology for collecting and transmitting data, in an attempt to reveal their potential impact in the field of agriculture. None of the literature highlights the application of AI techniques and robots in (Cultivation, Monitoring, and Harvesting) to understand their contribution to the agriculture sector and the simultaneous comparison of each based on its usefulness and popularity. This work investigates the comparative analysis of three essential phases of agriculture: Cultivation, Monitoring, and Harvesting, by knowing the depth of AI involved and the robots utilized. The current study presents a systematic review of more than 150 papers based on the existing automation application in agriculture from 1960 to 2021. It highlights the future research gap in making intelligent autonomous systems in agriculture. The paper concludes with tabular data and charts comparing the frequency of individual AI approaches for specific applications in the agriculture field.
A review of core agricultural robot technologies for crop productions
2023, Computers and Electronics in Agriculture
It has become a common understanding that robotization could be one of effective solutions to address the increasing skilled labor shortage challenge in crop production, thus the research and development of agricultural robotic technologies has become a popular topic in recent years, with numerous innovative new technologies, methodologies and applications being published at various resources. This article intends to provide a systematic overview of the recent achievements on core agricultural robotic technologies for crop productions, mainly on three aspects of (1) mobile platforms and in-field navigation technologies; (2) actuation mechanisms and end-effector technologies; and (3) target detection and operation planning technologies. This article also briefly introduces a few recent reported successes in developing robotic solutions for various field operations, such as transplanting, weeding, harvesting, and pruning, as evidences of promising.
Towards real-time tracking and counting of seedlings with a one-stage detector and optical flow
2022, Computers and Electronics in Agriculture
Citation Excerpt :
Object detection is an intuitive approach to count crops and fruits in static images. Researchers have developed image processing algorithms based on color, shape, and texture to detect broccoli seedlings (Zou et al., 2019), grapes (Behroozi-Khazaei and Maleki, 2017; Skrabanek and Doležel, 2017), bananas (Fu et al., 2019), and apples (Lv et al., 2016; Liu et al., 2019). Recently, deep learning methods, such as deep convolutional neural network (CNN) have created new opportunities for object detection because deep learning techniques can learn features automatically through supervised learning (Cheng et al., 2017).
The population of crop seedlings is important for breeders and growers to evaluate the emergence rate of different cultivars and the necessity of replanting, but manual counting of plant seedlings is time-consuming and tedious. Building upon our prior work, we advanced the cotton seedling tracking method by incorporating a one-stage object detection deep neural network and optical flow to improve tracking speed and counting accuracy. Videos of cotton seedlings were captured using consumer-grade video cameras from the top view. You Only Look Once Version 4 (YOLOv4), a one-stage object detection network, was trained to detect cotton seedlings in each frame and to generate bounding boxes. To associate the same seedlings between adjacent frames, an optical flow-based tracking method was adopted to estimate camera motions. By comparing the positions of bounding boxes predicted by optical flow and detected by the YOLOv4 network in the same frame, the number of cotton seedlings was updated. The trained YOLOv4 model achieved high accuracy under conditions of occlusions, blurry images, complex backgrounds, and extreme illuminations. The F₁ score of the final detection model was 0.98 and the average precision was 99.12%. Important tracking metrics were compared to evaluate the tracking performance. The Multiple-Object Tracking Accuracy (MOTA) and ID switch of the proposed tracking method were 72.8% and 0.1%, respectively. Counting results showed that the relative error of all testing videos was 3.13%. Compared with the Kalman filter and particle filter-based methods, our optical flow-based method generated fewer errors on testing videos because of higher accuracy of motion estimation. Compared with our previous work, the RMSE of the optical flow-based method decreased by 0.54 and the counting speed increased from 2.5 to 10.8 frames per second. The counting speed can reach 16.6 frames per second if the input resolution was reduced to 1280 × 720 pixels with an only 0.45% reduction in counting accuracy. The proposed method provides an automatic and near real-time tracking approach for counting of multiple cotton seedlings in video frames with improved speed and accuracy, which will benefit plant breeding and precision crop management.
Measuring radiata pine seedling morphological features using a machine vision system
2021, Computers and Electronics in Agriculture
Radiata pine seedlings are raised in nursery beds and are lifted and sorted in winter before being sold to forestry companies. This happens to order, as bare-root stock cannot survive for long out of the ground. The sorting criteria are usually clearly defined but exact measurements cannot be obtained due to the accuracy of the measuring equipment and the variability in the tree stock due to their organic nature. Machine vision is employed to measure the seedling features since it is more objective, faster, and less prone to error than manual measurements. This article presents three algorithms to measure the seedling features of root collar diameter, seedling height and root spread using a machine vision system with image processing techniques of noise removal and segmentation. The unnecessary information is removed by dilation and erosion while segmentation is based on the variances of the pixels. Processing time was approximately 30 ms for an entire seedling. The machine vision measurements are compared to the manual measurements. There is good correlation between the measurements of seedling features taken with the machine vision system and manual process: 95% of machine vision measurements for root collar diameter are within approximately 1.3 mm of manually measured values; 95% of machine vision measurements for height are within 18 mm of manual measurements; and 95% of machine vision measurements for maximum void angle of roots are within 87° less, and 78° more than manual measurements.

View all citing articles on Scopus

View full text

Original papersA robust algorithm based on color features for grape cluster segmentation

Highlights

Abstract

Introduction

Section snippets

Materials and methods

Results and discussion

Conclusions and future works

Appl. Soft Comput.

Comput. Electron. Agri.

Comput. Electron. Agric.

Comput. Electron. Agric.

Appl. Soft Comput.

Comput. Electron. Agric.

Comput. Electron. Agric.

Comput. Electron. Agric.

Food Bioprod. Process.

Control Eng. Pract.

Expert Syst. Appl.

Detection of cherry tree branches with full foliage in planar architecture for automated sweet-cherry harvesting

Biosys. Eng.

Combined application of decision tree and fuzzy logic techniques for intelligent grading of dried figs

J. Food Process Eng.

Grape clusters and foliage detection algorithms for autonomous selective vineyard sprayer

J. Intell. Service Robot.

Early yield prediction using image analysis of apple fruit and tree canopy features with neural networks

J. Imag.

Grapevine yield and leaf area estimation using supervised classification methodology on RGB images taken under field conditions

Sensors

Yield prediction from digital image analysis: a technique with potential for vineyard assessments prior to harvest

Aust. J. Grape Wine Res.

Original papers
A robust algorithm based on color features for grape cluster segmentation