Stereo-vision-based crop height estimation for agricultural robots

doi:10.1016/j.compag.2020.105937

Computers and Electronics in Agriculture

Volume 181, February 2021, 105937

https://doi.org/10.1016/j.compag.2020.105937 Get rights and content

Highlights

•
A stereo camera system captures accurate crop height measurements.
•
Disparity and depth maps determine the edges of regions of interest (ROI).
•
Crop regions are segmented using the edges without pre-labelling.
•
The system detects the target crop region even when objects overlapped in images.
•
Estimated crop heights showed strong linear correlations with actual crop heights.

Abstract

The objective of this study was to develop a machine-vision-based height measurement system for an autonomous cultivation robot. The system was developed with a simple stereo camera configuration to facilitate practical field applications and was used to acquire accurate height measurements of various field crops. The acquired stereo images were converted to disparity maps through stereo matching, and the disparity of each pixel was calculated to determine the depth of the distance between the camera and the crop. Depth maps were used to determine the edges of regions of interest (ROI) and the crop regions were segmented using the edges located in the expected crop region closest to the camera without using any additional labelling. The crop height was calculated using the highest points in the ROI. This approach was tested on five crops, and the results showed that the system could detect the target crop region even when objects overlapped in the acquired images. Furthermore, the crop heights estimated with the developed system showed strong agreement with actual crop heights measured manually, with the R² ranging from 0.78 to 0.84. These results indicate that the developed algorithm is capable of measuring crop heights in various ranges for agricultural robot applications.

Introduction

The global agricultural robot market was estimated to be $4.1 billion in 2017 and is expected to increase steadily to approximately $25.2 billion in 2025 (Market research report, 2019). Market growth in this sector has accelerated because of recent agricultural issues, such as aging-related reductions in the labor force and a projected shortfall in the amount of food available for the global population in 2050. These issues have pushed the agricultural industry to develop advanced autonomous farming products to reduce the labor force, enhance productivity, and manage crops more efficiently. Specifically, automatic monitoring or scouting robots that can survey crop growth without human resources have been studied extensively because crop growth monitoring is among the most frequent and time-consuming processes in farm operations (Chou et al., 2019).

Crop height is an especially important phenotypic factor that indicates overall crop growth and is used to predict yield and manage crop cultivation. Crop height is usually observed manually, which is labor-intensive and costly (Hu et al., 2018); hence, several approaches have tried to measure crop height automatically with various sensors. For instance, Zhang and Grift (2012) developed a measurement system for the stem height of Miscanthus giganteus based on Light Detection and Ranging (LiDAR). The accuracy of the developed system was over 92% in terms of crop density and was reported to achieve real-time crop height evaluation. Although the system can collect 3D structural data accurately, the LiDAR used in this work is costly and has portability limitations for in-field farming machines. Ultrasonic sensor-based approaches that is more economical than LiDAR have also been conducted. Chang et al. (2017) developed a robot-based system using an ultrasonic-based plant height measurement system to control the head position of a harvester during blueberry harvesting. The first prototype of the system could estimate blueberry height within an error of 5.7 cm, and the second prototype reduced the error to 1.7 cm. They showed remarkable results; however, the accuracy of ultrasonic sensors is affected by the material’s softness, and such measurements are challenging when objects are covered in an extremely soft fabric that absorbs sound waves, as is often the case with crops. Ultrasonic waves are also unreliable in agriculture because of their sensitivity to temperature and wind and their associated impacts on the measurements over time.

In the last decade, vision-based technologies have received increasing attention because they can be tag-free, offer inexpensive configurations (Carrasco, 2011), and can provide 3D structural information. 3D image-based studies targeting object recognition in agriculture have been conducted with stereo vision and depth cameras (e.g., Microsoft Kinect). For example, 3D structure estimations have been applied to plant phenotyping (Andújar et al., 2016, Jiang et al., 2016, Malekabadi et al., 2019) and crop row mapping (Jay et al., 2015). Among studies specifically targeting crop height, Jiang et al. (2016) developed a tractor-attached system to measure the height of cotton plants using a Kinect-v2, and the results showed that the system could measure the height under real-world field conditions. Andújar et al. (2016) constructed a 3D model of cauliflower plants using a Kinect-v1, and the developed algorithm showed only a 2-cm deviation compared to the actual height. Malekabadi et al. (2019) obtained stereo images and calculated disparity maps of trees by foliage density and canopy shape, and the developed algorithm could calculate height with errors of less than 7% for both elliptical and conical trees. However, although these studies have advanced crop height estimation technology, they were conducted for only one type of crop and in a specific environment or laboratory condition with controlled light intensity. In particular, Kinect uses structured-light (v1) or time-of-flight (v2) based depth estimation, and the use of infrared radiation range wavelengths makes it difficult to implement under natural light conditions.

Our purpose is to estimate crop heights under uncontrolled field conditions, and the target task is focused on cultivation, not phenotyping. The aforementioned studies were generally performed for phenotypic analyses with individual small sized crops, and these approaches have limitations when applied to automatic cultivation. Images were acquired using cameras oriented towards the nadir with a stable background, and the region of interest (ROI) in each work was predetermined using GPS or markers. The former is not practical for a forward view of automatic cultivation; the latter is cumbersome preliminary work, making end-to-end automation difficult. For use in practical cultivation, such a system must be able to acquire crop images in the forward travel direction and to distinguish crop regions from backgrounds such as soil, weeds, and other crops with similar phenotypes, without additional labelling. Therefore, with the final goal of designing an agricultural robot for cultivation, this study focused on stereo vision-based crop height estimation of various upland crop types under uncontrolled field conditions, with images acquired in the forward direction and gathered from overlapping crops at various distances. Our system can provide the representative height of a target crop region automatically in an end-to-end process during crop cultivation, and it can also contribute to realizing precision agriculture by real-time crop height monitoring with spatial matching.

Section snippets

Stereo image matching

Stereo cameras used to acquire crop images have binocular stereo vision, as illustrated in Fig. 1, implemented by two image sensors with parallel directions. Thus, the distance between the camera and the object can be calculated by Eq. (1), where the distance of the target point (p) projected on the left ( $x_{L}$ ) and right ( $x_{R}$ ) image sensors is calculated with depth information acquired from the disparities between the two images in the horizontal (x-axis) pixel distance of the target point (Adi

Results of stereo matching

The representative disparity maps for each crop are shown in Fig. 6. The disparity maps are shown in gray scale, and the intensity of each pixel depends on the distance, such that the closer to white the color is, the closer the object is to the camera. The black band of the disparity map is a region where the left and the right image do not match due to the horizontal shift between images.

The Chinese cabbage images showed high-density crop regions in which the leaf area was larger and leaves

Conclusion

In this study, stereo-vision-based automatic crop height estimation was demonstrated with a system that can be practically applied in an agricultural robot or a conventional farm machine. The developed system has a simple configuration that includes a stereo camera and a single board computer. This system does not require preliminary labelling, and it was shown to successfully measure the height of field crops in front of the machinery on which it was mounted during cultivation. The proposed

CRediT authorship contribution statement

Wan-Soo Kim: Investigation, Validation, Writing - original draft, Writing - review & editing. Dae-Hyun Lee: Conceptualization, Software, Supervision, Writing - original draft, Writing - review & editing. Yong-Joo Kim: Funding acquisition, Project administration. Taehyeong Kim: Conceptualization, Software. Won-Suk Lee: Conceptualization, Writing - review & editing. Chang-Hyun Choi: Conceptualization, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research was supported by the Agricultural, Forestry, Food Research Center Support Program (Project No.: 714002-07), Ministry of Agricultural, Food and Rural Affairs. It was also supported by the Korea Institute of Planning and Evaluation for Technology in Food, Agriculture, Forestry (IPET) through the Advanced Production Technology Development Program, funded by the Ministry of Agriculture, Food and Rural Affairs (MAFRA) (319041-03).

References (17)

D. Andújar et al.
Using depth cameras to extract structural parameters to assess the growth state and yield of cauliflower crops
Comput. Electron. Agric.
(2016)
M. Carrasco
Visual attention: The past 25 years
Vision Res.
(2011)
Y.K. Chang et al.
A real-time ultrasonic system to measure wild blueberry plant height during harvesting
Biosyst. Eng.
(2017)
W.C. Chou et al.
Prioritization of pesticides in crops with a semi-quantitative risk ranking method for Taiwan postmarket monitoring program
J. Food Drug Anal.
(2019)
L. He et al.
The connected-component labeling problem: A review of state-of-the-art algorithms
Pattern Recognit.
(2017)
P. Hu et al.
Estimation of plant height using a high throughput phenotyping platform based on unmanned aerial vehicle and self-calibration: Example for sorghum breeding
Eur. J. Agron.
(2018)
S. Jay et al.
In-field crop row phenotyping from 3D modeling performed using structure from motion
Comput. Electron. Agric.
(2015)
Y. Jiang et al.
High throughput phenotyping of cotton plant height using depth images under field conditions
Comput. Electron. Agric.
(2016)

There are more references available in the full text version of this article.

Cited by (68)

Machine vision based plant height estimation for protected crop facilities
2024, Computers and Electronics in Agriculture
The increasing demand for quality, year-round food production in limited space has led to the widespread adoption of protected cropping. Effectively monitoring and maintaining crops within these facilities requires substantial labour and expertise. Traditional manual monitoring is labour intensive and time consuming. Therefore, non-destructive image-based techniques, particularly those utilising 3D structural data, have gained attention. We developed a stereo vision-based system to estimate the height of vertically supported tall plants in protected facilities, given plant height serves as a vital measure of crop growth. Our system uses a mobile platform with a top-angle view of a stereo vision depth camera for data acquisition and machine learning in its core for data analysis. First, we collected weekly RGB and depth (RGBD) streams from plant gutters in three glasshouse compartments with different light treatments. We used part of the RGB data collected to train and validate a deep learning segmentation model to detect plant tops and bases. Detected tops and bases of an image were then mapped to the generated 3D scene using the depth image of the same frame. Thresholds and 3D clustering are used respectively to remove background and eliminate outliers in top and base detection mapped to 3D space. Finally, the height of each plant was calculated using the cluster centres of the tops and bases of the plants. Manually measured heights of ten selected plants per environment were used to validate the height estimations. Similar growing patterns were observed between imaged and manually measured plant heights, which showed strong correlations of 0.87, 0.96, and 0.79 $R^{2}$ scores, respectively, under unfiltered ambient light, Smart Glass film, and shifted light. These promising results demonstrate the feasibility of our proposed method for a vertically supported capsicum crop in a commercial-scale protected crop facility.
OrangeStereo: A navel orange stereo matching network for 3D surface reconstruction
2024, Computers and Electronics in Agriculture
Three-dimensional (3D) surface reconstruction is crucial to accurately detecting navel orange external qualities such as size and shape, and binocular stereo vision is promising for practical sorting scenarios due to its high frame rates and ease of deployment. For objects with texture-less surface, such as navel oranges, it is very challenging to match the corresponding points using traditional stereo matching algorithms because the detected feature points have high similarity in color as well as structure. Most of the existing deep learning-based stereo matching algorithms focus on the stereo matching performance of the whole image and lack the attention to the matching performance of the main region. Meanwhile, simple background with less texture will bring more uncertainty to the matching of corresponding points, which affects the robustness of the model. The OrangeStereo was proposed based on the GwcNet structure to achieve a better corresponding points matching effect of navel oranges, which can provide more accurate 3D surface information. The Structural Features Extraction (SFE) module was introduced after the initial feature extraction module to extract stable structural features and avoid the limitations of using single convolutional features. The Attention Weights Generation (AWG) module was utilized at matching cost calculation stage to help suppress redundant information and make the network pay more attention to the matching performance of specific regions. The improved loss function assigned a greater weight to the loss of the navel orange region using semantic information from left image to make the network more focused on the matching effect of the navel orange region. An ablation study was conducted to confirm the efficiency of the SFE module, AWG module, and the improved loss function of the algorithm. The ablation study contained five sub-experiments, including the baseline model, adding the SFE and the AWG modules respectively, and whether to add the improved loss function based on the addition of previous two modules. Additionally, PSMNet, GwcNet, RealtimeStereo, and ACVNet, four typical stereo matching methods, were contrasted with the OrangeStereo. The proposed OrangeStereo achieved better performance with EPE, RMSE, bad-1, bad-3, bad-5, and inference time of 0.60 pixels, 3.53 pixels, 2.37 %, 1.83 %, 1.62 %, and 33 ms in the orange region, respectively. The R² and RMSE for the depth of orange utilizing our method were 0.982 and 0.81 mm, respectively. The proposed algorithm can be employed to obtain accurate depth information of fruits quickly, and it is expected to be used in the 3D reconstruction of fruits in the actual commercial fruit sorting lines to achieve more accurate external quality assessment.
Semantic surrounding projection for object height estimation using single-line lidar and image
2024, Computers and Electrical Engineering
Indoor environment perception requires surroundings to facilitate the path planning and object management using three dimensional lidar or visual sensors,which addresses the need for accurate height estimation in tasks like estimating cabinet height for grabbing things and judging passageway passing height. The proposed method combines single-line lidar and image to extract features, such as line segmentation, objects detection and semantic segmentation map, using neural network. Height estimation is achieved by referencing single image with single-line lidar data. The multi-source data approach enhances the accuracy especially in the areas where data from a single sensor are unreliable. A set of analysis of errors and uncertainties introduced by the method are conducted to facilitate the indoor perception system through the integration of muti-source data and neural network-base feature extraction for height estimation, and results demonstrate the performance and efficiency of the method.
A method for obtaining maize phenotypic parameters based on improved QuickShift algorithm
2023, Computers and Electronics in Agriculture
Phenotype is an important bridge for studying the mechanism of “genotype-phenotype-environment”, and studying precise measurement of crop individual phenotypic information is of great significance for accelerating breeding processes and assisting in precision agriculture monitoring. The purpose of this study is to develop a clustering algorithm for corn population point clouds to accurately extract the three-dimensional morphology of individual crops. Firstly, this study used drones to obtain field corn data, reconstructed corn plant models based on the SfM algorithm, simplified and rotated point clouds, and extracted crop point clouds using methods based on color and distance thresholds. For density segmentation, this study improved the QuickShift method to segment individual corn crops. Color weights were added to the clustering density calculation process, and the original point cloud was projected onto a horizontal plane to calculate the density clustering core. Then, the remaining points were subjected to QuickShift clustering segmentation in 3D to obtain the point cloud data of individual corn plants. The use of cylindrical fitting and conditional Euclidean clustering algorithms to automatically segment point clouds of stems and leaves has achieved estimation of plant height, stem thickness, minimum bounding box volume per plant, and number of leaves. Finally, the true and measured values of plant height and stem thickness were compared. The results showed that the accuracy of the improved QuickShift method for segmentation was 93.10%, with an average measured plant height of 14.21 cm and an average stem diameter of 0.72 cm. The determination coefficient and root mean square error of the automatic and manual measurement results of plant height are 0.9186 and 0.5785 cm, respectively. The determination coefficients and root mean square errors of the automatic and manual measurement results of stem thickness are 0.8227 and 0.0574 cm, respectively. This study uses an improved QuickShift to cluster and segment corn point cloud data, achieving the segmentation of individual corn plants in the field, providing an automated, efficient, and low-cost solution for accurately measuring crop phenotypic information.
Using structure-from-motion to estimate cover crop biomass and characterize canopy structure
2023, Field Crops Research
Variability in biomass production poses a challenge for growers when using cover crops for weed control. However, most methods for assessing cover crop biomass are laborious and impractical on a field scale.
The goal of the present study was to determine the feasibility of using Structure-from-Motion (SfM) photogrammetry to estimate biomass in cereal rye (Secale cereale L.) and winter wheat (Triticum aestivum L.) cover crops by correlating biomass with 3-D point cloud pixel density and crop height.
Point clouds were generated using a SfM algorithm from RGB (red, green, and blue) videos collected by a hand-held GoPro camera over sixteen crop fields in North Carolina, Iowa, and Maryland, USA, throughout two growing seasons (2021–2023). Crop height, leaf area index (LAI), and photosynthetically active radiation (PAR) were also measured.
Biomass was positively correlated with crop height for both cereal rye (R² = 0.621) and wheat (R² = 0.55). LAI was positively correlated with biomass accumulation and crop height for both species, increasing linearly in rye and exponentially in wheat. Conversely, PAR penetration below the canopy decreased with biomass accumulation and crop height in both species, with a more rapid extinction in wheat than rye. Point cloud pixel density showed a positive linear relationship with biomass in rye but saturated after 2.5 tonnes ha⁻¹ (2500 kg ha⁻¹). In wheat, point cloud pixel density was weakly and negatively correlated with biomass due to a denser canopy causing faster saturation of tissue detection by SfM point clouds. However, considering crop height and point cloud density integrating them both in the model allowed obtaining a positive relationship with biomass through levels of 8 tonnes ha⁻¹ (8000 kg ha⁻¹) in both species. When models were validated with independent data, predicted and measured biomass were positively correlated for both rye (R² = 0.86) and wheat (R² = 0.78).
Based on the results, using SfM to generate 3-D point clouds can provide a more accurate estimation of biomass than canopy height alone by capturing species-level differences in canopy architecture.
The results of this study suggest that SfM can potentially be used as a non-destructive tool for growers to monitor biomass production in cereal cover crops other systems such as energy/forage crops, which can help inform management decisions and conserve resources.
How artificial intelligence uses to achieve the agriculture sustainability: Systematic review
2023, Artificial Intelligence in Agriculture
The generation of food production that meets the rising demand for food and ecosystem security is a big challenge. With the development of Artificial Intelligence (AI) models, there is a growing need to use them to achieve sustainable agriculture. The continuous enhancement of AI in agriculture, researchers have proposed many models in agriculture functions such as prediction,weed control, resource management, advance care of crops, and so on. This article evaluates on a systematic review of AI models in agriculture functions. It also reviews how AI models are used in identified sustainable objectives. Through this extensive review, this paper discusses considerations and limitations for building the next generation of sustainable agriculture using AI.

View all citing articles on Scopus

View full text

Stereo-vision-based crop height estimation for agricultural robots

Highlights

Abstract

Introduction

Section snippets

Stereo image matching

Results of stereo matching

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgements

Comput. Electron. Agric.

Vision Res.

Biosyst. Eng.

J. Food Drug Anal.

Pattern Recognit.

Eur. J. Agron.

Comput. Electron. Agric.

Comput. Electron. Agric.