Machine vision for orchard navigation
Introduction
Within the next 30 years, the world’s population is preparing for a steep increase, bracing for nearly 10 billion inhabitants [1]. With a vastly growing population, the demand for natural food resources needed to sustain this population expands simultaneously. However, with the labor force moving away from manual agriculture, farmers are now needing to develop ways to harvest and produce an increased output of product while harboring the loss of physical manpower [2].
As technology increases, researchers are beginning to implement robotics in the field, creating sustainability for farmers amid declining labor supply and increasing food demand. In addition to using robotics technology, a farmer can also use “precision agriculture” [3]: a way to create efficient cultivation by maximizing production while minimizing costs, which is very compatible with robotics [4]. While in the past, many agriculturalists have been focused on a large output of goods to feed the growing population, currently much of the population is becoming more concerned with how these goods are being produced. People, as well as governments, are caring more and more about the chemicals used in food products, as well as the methods implemented for seeding and harvest. Also, the public and farmers alike are beginning to be concerned with the environmental impacts of agriculture, such as synthetically enhancing crops, harsh chemical weed control, and precision watering. With these needs considered, robots, especially autonomously driven robots, can help the agricultural community achieve these newfound regulations and concerns. By using an autonomous robot, farmers can reduce the manpower and hours needed to do these mundane tasks—like precision weed spraying, seed or row mapping, precision watering, and harvest data gathering—yet still carry out these undertakings precisely. Farmers can also conduct other high-level tasks, as navigation is automated [5]. Other benefits can include increased safety of the harvest and equipment, longer duration of work, as an autonomous vehicle may outlast a human worker, as well as increased productivity [6]. Pederson et al. describes the viability of robotics in the field, by studying vehicles doing three different autonomous tasks in an agricultural setting, and finding that these robots are more economically friendly than the traditional methods of accomplishing the same assignment [7]. By implementing autonomous robots into an agricultural setting, farmers can better use their resources, therefore saving money and economizing a better product for the masses. With these benefits in mind, no longer are autonomous vehicles in the field ideas for the future, but they are becoming feasible, and much needed, option for today [8].
The main challenge for developing an autonomous vehicle on the farm is its navigation system. Navigating a vehicle in a field with different crops is not a trivial matter. One vital vehicle on the farm is the tractor. By using a tractor, a farmer can cultivate a wide area at once with a large amount of power. The idea of “driverless tractors” stems back to the 1920’s, and gained further momentum in the 1950’s and 1960’s, with prototypes being guided by large cables in field rows. In the 1980’s and 1990’s, research for machine vision and computer aid guidance were implemented through research done through Michigan State and Texas A&M universities, as well as automated fruit harvesting at the University of Florida [9]. This research has continued into present North America, with more funding being poured into the agricultural domain, gaining attention from prestigious research academies and personnel looking into various sensors and machines [10]. However, since precision is beginning to outweigh brute force, other types of machinery are also being considered. In some cases, a tractor can seem unreliable for tasks such as turning into narrow rows, or accurate weed spraying or watering. Likewise, a navigational system that may work for a large machine can prove inefficient for a smaller one, which may be needed for precision farming. Another inefficiency with some of this research is that it is often conducted in a controlled laboratory environment, which differs greatly from the outdoor setting for which it would be used [11].
One of the main tasks of autonomous guidance is developing a functional navigation system. The navigation of an autonomously driven vehicle can prove a difficult matter. With different methods, such as global positioning systems, mechanical feelers, dead reckoning, radio frequency identification tags, laser radar, Kalman filter, and machine vision, one has yet to be named the better, as each one has its difficulties [11]. In this paper, a machine vision sensor is proposed to guide a vehicle in a row of trees. The basic concept of machine vision for navigation is the machine vision sensor can measure the relative position of the vehicle concerning a landmark and use that to estimate the vehicle’s heading. Usually, the vision sensor is mounted on the vehicle [9]. The position of the vision sensor is dependent on the geometric relationship between the sensor and vehicle, and the field of view of the sensor. One of the common methods of using landmarks is detecting a row of crops on the ground. Benson et al. developed a machine vision system to guide a combine harvester, and it used the lateral position of the crop cut edge as a guidance directrix [12]. One of the difficulties in developing a machine vision for outdoor application is dealing with the varying ambient light condition. A machine vision guidance system for a paddy rice field used a pair of monochrome cameras with different spectral filters to deal with strong reflections on the water [13]. Hague et al. differentiated between vegetation and soil using a near-infrared camera to enhance the high reflectance of plants in the near infrared [11]. In addition to using special filters, different image processing techniques are also explored to take advantage of the geometrical properties of the crops. For example, Tillet and Hague [14] developed a machine vision guidance system for cereal crops using the midpoints of rows extracted from a single view of adjacent row crops. Sogaard and Olsen [15] used a color video camera to detect and localize small-grain crops and calculated the centers of gravity of row segments without using segmentation. Other researchers also used a Hough transform to track rows of transplanted cauliflower field [16].
Much of the current research in machine vision concentrates on ground-based feature recognition solutions, such as processing images of the surface crop, grass edge, planting pattern, or other low landmarks to find a guidance directrix. This is a logical approach specifically for surface crops because they are planted in a certain pattern, and the features of the pattern can be used to guide the vehicle. On the other hand, orchards with fruit trees pose a different challenge. While some researchers have used the overhanging canopy to use as a guide, it calls for a different approach to deal with the challenges. Subramanian et al. [17] developed a machine vision system for citrus groves, and it used adaptive thresholding to deal with the changing illumination and shadows brought about by the canopy. While an agriculture setting, specifically orchards, can prove to be one of the ideal settings for this kind of research due to the environment in which it resides, there are always challenges in the field [18]. In an orchard, the trees and rows are unlikely to change from year to year, yet the trees or vines themselves change from blossoms to fruit throughout the season. Along with the changing trees (changing in color, growth, and foliage), an orchard can have narrow rows and smaller turning radii, as well as uneven terrain with obstacles, such as branches from trimming or else tarps for collecting [19]. Changing settings like this lead to challenges with navigation, especially if that navigation relies on physical barriers as indicators. Along with adapting to these constraints, researchers must develop algorithms that adapt to the season, yet that also remain low cost and simple for farmers to use.
When relying on ground features, they are often subject to change, due to varying environmental factors and the dynamic nature of the crops. This research focuses on looking up, not down, with the camera angles and uses features of the sky and tree canopies in the image processing. By focusing on the tree canopies and sky, the environmental factors decrease drastically. Unlike the trees themselves, the skylines do not change throughout the seasons, and always remains as one of the most luminescent parts of the image, a key in pinpointing an object in image processing. By using the sky as a guidance system, this research provides a unique approach to autonomous navigation using machine vision.
This paper is aimed at finding ways to develop and implement machine vision into the agricultural field, predominantly orchards. The goal of this research is to develop an algorithm to autonomously guide a small robotic ground vehicle platform along an orchard row, following the path of the row using an upward looking camera combined with a controller based on feature recognition from the contrast between the tree canopies and the sky. The autonomous navigation system will be evaluated in laboratory controlled settings and actual field setting using a commercial peach orchard.
Section snippets
Material and methods
The block diagram of the unmanned ground vehicle (UGV) navigation system (Fig. 1) has three main components: the unmanned ground vehicle platform, the visual feedback system, and the controller. The input to the system is the desired vehicle position, and the visual feedback system is used to estimate the current vehicle position. The controller to correct the position of the vehicle as it moves along the tree rows uses the difference between the desired position and the current position.
Lab test results
As mentioned above, the image processing algorithm and vision control system was first tested on a straight sidewalk. Regarding the image processing, testing would be very similar to an orchard and a sidewalk with the only differences being the camera angle-pointing towards the ground for the sidewalk tests and towards the sky for orchard tests. The basic image processing algorithm was identical except the path plane was not inverted, locating the centroid of the binary image and using it as
Discussion
Based on the test results, it was determined that the image-processing algorithm for the vehicle guidance system was adequate for guiding the vehicle down the orchard row. The difficulties such as inconsistent lighting, shadows, and color similarities in features were eliminated by using the sky-based approach where the image was reduced to canopy plus sky, thus simplifying the segmentation process. Feature detection using machine vision is facilitated if the segmentation process is effective.
Conclusion
A machine vision system to guide a small-unmanned ground vehicle system in an orchard was developed. The vehicle is differentially steered by four DC motors. The navigation is based on a visual servo system using machine vision system, which is composed of a multispectral camera to acquire images and a personal computer to process the images. The camera was positioned to focus on the tree canopies and the sky, which is a sky-based approach. The image processing segmented the sky from the tree
Acknowledgment
This research was supported by the Idaho Space Grant Consortium, NASA EPSCoR, and Northwest Nazarene University.
Glossary
- Autonomous navigation
- Operation of a vehicle system without human intervention
- Machine vision
- A sensor system that uses an image acquisition system like a camera and image processing to extract features for process control
- Precision agriculture
- A site-specific technology of observing field variability and responding to this variability to optimize crop production
References (19)
- et al.
Sensing technologies for precision specialty crop production
Comput. Electron. Agric.
(2010) A technical review on navigation systems of agricultural autonomous off-road vehicles
J. Terramech.
(2013)- et al.
A guidance directrix approach to vision-based vehicle guidance systems
Comput. Electron. Agric.
(2004) - et al.
Agricultural automatic guidance research in North America
Comput. Electron. Agric.
(2000) - et al.
Machine vision-based guidance system for agricultural grain harvesters using cut-edge detection
Biosyst. Eng.
(2003) - et al.
A dual-spectral camera system for paddy rice seedling row detection
Comput. Electron. Agric.
(2008) - et al.
Determination of crops rows by image analysis without segmentation
Comput. Electron. Agric.
(2003) - et al.
Real-time tracking of plant rows using a Hough Transform
Real-Time Imaging
(1995) - et al.
Development of machine vision and laser radar based autonomous vehicle guidance systems for citrus grove navigation
Comput. Electron. Agric.
(2006)
Cited by (114)
An improved target detection method based on YOLOv5 in natural orchard environments
2024, Computers and Electronics in AgricultureVision-based autonomous navigation stack for tractors operating in peach orchards
2024, Computers and Electronics in AgricultureNavigation path recognition between rows of fruit trees based on semantic segmentation
2024, Computers and Electronics in AgricultureTowards autonomous mapping in agriculture: A review of supportive technologies for ground robotics
2023, Robotics and Autonomous SystemsResearch progress of autonomous navigation technology for multi-agricultural scenes
2023, Computers and Electronics in AgricultureStudy of convolutional neural network-based semantic segmentation methods on edge intelligence devices for field agricultural robot navigation line extraction
2023, Computers and Electronics in Agriculture