A framework for the management of agricultural resources with automated aerial imagery detection

Highlights

• Efficient pipeline for localizing, transporting and managing tropical fruits.

• Fast agricultural resources and infrastructure detection from aerial imagery.

• Density map production for all types of fruits based on the detection result.

• Resources collection path optimization based on specific scenarios and objectives.

Abstract

The acquisition of data through remote sensing represents a significant advantage in agriculture, as it allows researchers to perform faster and cheaper inspections over large areas. Currently, extensive researches have been done on technical solutions that can benefit simultaneously from both: vast amounts of raw data (big data) extracted from satellite images and Unmanned Aerial Vehicle (UAV) and novel algorithms in Machine Learning for image processing. In this experiment, we provide an approach that fulfills the necessities of rapid food security, assessment, planning, exploitation, and management of agricultural resources by introducing a pipeline for the automatic localization and classification of four types of fruit trees (coconut, banana, mango, and papaya) and the segmentation of roads in the Kingdom of Tonga, using high-resolution aerial imagery (0.04 m).

We used two supervised deep convolutional neural network (CNN): the first, to localize and classify trees (localization) and the second, to mask the streets from the aerial imagery for transportation purposes (semantic segmentation). Additionally, we propose auxiliary methods to determine the density of groupings of each of these trees species, based on the detection results from the localization task and render it in Density Maps that allow comprehending the condition of the agriculture site quickly. Ultimately, we introduce a method to optimize the harvesting of fruits, based on specific sceneries, such as maximum time, path length, and location of warehouses and security points.

Introduction

Located in the Pacific Ocean, the Kingdom of Tonga extends over an area of 362,000 km². With a population of 107.122 inhabitants in 2016, 58.4% of its population depends on agriculture and forestry as a primary source of income and a key driver for economic growth. Its most prominent agricultural products are bananas, coconuts, coffee beans, vanilla beans, and roots such as cassava, sweet potato, and taro² (Halavatau and Halavatau, 2001).

Most of the countries in the Pacific region are exposed to high-risk disasters including cyclones, earthquakes, tsunami, storm surge, volcanic eruptions, landslides, and droughts, e.g., Tonga is affected by more than one tropical cyclone every four years. Theses recurrent disasters cause damage and losses to agriculture, food security and local economy. In the last years, according to the 2015 Report of the Secretary-General on the Implementation of the International Strategy for Disaster Reduction; disasters worldwide cost around USD 1.5 trillion in economic damage. The frequency and severity of natural disasters are increasing, revealing an urgent need to strengthen the resilience of food assessments and security (FAO, 2015).

To understand how local agriculture and food security were affected by a natural disaster, aerial imagery from the site and the succeeding mapping and classification of data are required. The field of Remote Sensing over the past decades has robustly investigated faster methods to collect, produce, classify, and map earth observation data. In recent years, the use of Unmanned Aerial Vehicles (UAV) to collect data has increased rapidly, mainly for their inexpensive hardware and rapidly deploy for the collection of imagery. In parallel, the development of new technics to detect objects in optical remote sensing imagery were actively explored³ by several scholars. In 1991 an automatic tree detection and delineation from digital imagery was performed by Pinz (1991) who proposed a Vision Expert System using aerial imagery. He was able to locate the center of trees crown and estimate their radius using local brightness maxima. In 1995 Gougeon (1995), launched a rule-based algorithm, that followed the valleys of shadows between tree crow in a ground sampled distance from digital aerial imagery. Hung et al. (2006) proposed a vision-based shadow algorithm for tree crowns to detect and classify imagery from UAV, using color and texture information to segment regions of interest. Hassaan et al. (2016) presented an algorithm to count trees in urban environments using image processing techniques for vegetation segmentation and tree counting. By applying a k-means clustering algorithm and setting threshold values to green clusters centers, the algorithm was able to segment out the green portion out of any image without any noise.

Today, the development of machine learning approaches provides researchers with a conceptual alternative to solve problems in the mentioned domains without predefining the rules for a specific task. Instead, models can learn the underlying features emerging from a large amount of data. One of the most prominent approaches comes from the field of image processing and computer vision named Convolutional Neural Network (CNN). The algorithm is based on an end-to-end learning process, from raw data to semantic labels, which is an essential advantage in comparison with previous state-of-the-art methods (Nogueira et al., 2017). This model outperforms all the other approaches in tasks like image classification, object recognition and localization, and pixel-wise semantic labeling. The early implementation of CNN by LeCun et al. (1998) achieved 99.2% of accuracy in handwriting digits recognition and led the boost of CNN based image processing in the following 20 years. In recent years, large online image repositories such as ImageNet (Deng et al., 2009), and high-performance computing platforms like GPU acceleration, have contributed significantly to the success of using CNN in a large-scale image and video recognition. Competitions and challenges like the ImageNet Challenge (Russakovsky et al., 2015) and Visual Object Classes Challenge (Everingham et al., 2015) attract many researchers and as a result, state-of-art CNN models such as AlexNet (Krizhevsky et al., 2012) and VGG-Net (Simonyan & Zisserman, 2014) respectively – both available online.

Moreover, researchers can directly use or train these models on their dataset with no need to design its architecture, e.g., YOLO model (Redmon et al., 2016) achieved an excellent performance on recognition and made the real-time object localization possible. In the meantime, Long et al. (2015) with their novel model FCN achieved 20% relative improvement in pixel-wise semantic segmentation in the PASCAL VOC challenge. Also, SegNet proposed by Badrinarayanan et al. (2015) also achieved competitive performance as it is designed to be efficient both in terms of memory and computational time during prediction – It is also significantly smaller in the number of trainable parameters than other competing architectures.

The use of deep learning⁴ in Remote Sensing has grown exponentially since it can effectively encode spectral and spatial information based on the data itself. During the last years, considerable efforts have been made to develop various methods for the detection of different types of objects in satellite and aerial images with CNN, such as road, vegetation, tree, water, buildings, cars, etc. – In the Conclusion section we address quantitative measures to support the effectiveness of the proposed approach compared to existing approaches: Chen et al., 2014, Luus et al., 2015, Lu et al., 2017, Kussul et al., 2017, Mortensen et al., 2016, Sørensen et al., 2017, Milioto et al., 2017.

In this paper, we aim to provide an approach that fulfills the necessities of rapid food security, assessment, planning, exploitation, and management of agricultural resources; we propose a framework to efficiently localize and classify four types of tropical fruits (coconut, banana, mango, and papaya). We pursue the latter by a method to automatically identify and segment roads, so that fastest and safest ways to transport crops to adjacent warehouses or security points can be detected.

To do so, we used two supervised deep CNNs; the first CNN model performs the task of object localization, to localize and classify the type of trees. The locations of the trees are not only used to control agricultural resources, but also in scenarios of natural disasters they can be compared with the previous state to have a better understanding on how local agriculture and food security were affected. This information can directly inform and accelerate subsequent relief efforts. Additionally, we propose a method to determine the density of each of these trees to improve productivity, based on the detection results of the first CNN and presented as Density Maps to quickly comprehend the condition of the agricultural site.

The second CNN model performs a semantic segmentation, that masks the streets from the aerial imagery to help identify local transportation infrastructure and, in the scenario of natural disasters, evaluates the damage, proposing a proper plan to distribute aid across affected areas. Ultimately, we introduce a method to optimize the harvesting process, based in specific sceneries, such as maximum time, path length, and location of the warehouse and security points.