An intelligent system for crop identification and classification from UAV images using conjugated dense convolutional neural network

Highlights

• Unmanned Aerial Vehicle (UAV) images are useful for crop classification.

• Authors propose new algorithm to identify crops from UAV images.

• Convolutional Neural Networks (CNN) do proper multi-class classification.

• Authors designed conjugated dense CNN (CD-CNN) for classifying five crops.

• CD-CNN outperforms other state-of-the art techniques in terms of accuracy.

Abstract

Crop identification and classification is an important aspect for modern agricultural sector. With development of unmanned aerial vehicle (UAV) systems, crop identification from RGB images is experiencing a paradigm shift from conventional image processing techniques to deep learning strategies because of successful breakthrough in convolutional neural networks (CNNs). UAV images are quite trustworthy to identify different crops due to its higher spatial resolution. For precision agriculture crop identification is the primal criteria. Identifying a specific type of crop in a land is essential for performing proper farming and that also helps to estimate the net yield production of a particular crop. Previous works are limited to identify a single crop from the RGB images captured by UAVs and have not explored the chance of multi-crop classification by implementing deep learning techniques. Multi crop identification tool is highly needed as designing separate tool for each type of crop is a cumbersome job, but if a tool can successfully differentiate multiple crops then that will be helpful for the agro experts. In contrast with the previous existing techniques, this article elucidates a new conjugated dense CNN (CD-CNN) architecture with a new activation function named SL-ReLU for intelligent classification of multiple crops from RGB images captured by UAV. CD-CNN integrates data fusion and feature map extraction in conjunction with classification process. Initially a dense block architecture is proposed with a new activation function, called SL-ReLU, associated with the convolution operation to mitigate the chance of unbounded convolved output and gradient explosion. Dense block architecture concatenates all the previous layer features for determining the new features. This reduces the chance of losing important features due to deepening of the CNN module. Later, two dense blocks are conjugated with the help of a conversion block for obtaining better performance. Unlike traditional CNN, CD-CNN omits the use of fully connected layer and that reduces the chance of feature loss due to random weight initialization. The proposed CD-CNN achieves a strong distinguishing capability from several classes of crops. Raw UAV images of five different crops are captured from different parts of India and then small candidate crop regions are extracted from the raw images with the help of Arc GIS 10.3.1 software and then the candidate regions are fed to CD-CNN for proper training purpose. Experimental results show that the proposed module can achieve an accuracy of 96.2% for the concerned data. Further, superiority of the proposed network is established after comparing with other machine learning techniques viz. RF-200 and SVM, and standard CNN architectures viz. AlexNet, VGG-16, VGG-19 and ResNet-50.

Introduction

Sustainable development in agriculture is highly required for precise farming. Proper identification and classification of crops is the fundamental and crucial step in precision agriculture before addressing other aims like plant counting, weed detection, health monitoring etc. Conventionally, skilled officials visit sites to identify different crops produced at a concerned area, which is quite laborious and time consuming task. In order to reduce the effort, in recent time aerial images of crop lands are utilized and processed. Crop plant identification from aerial view of land require high spatial resolution images in order of few cm or mm (Hengl, 2006). Satellites like GeoEye-1 and Worldview-2 are utilized for acquiring high resolution images of earth surface with spatial resolution lesser than 1meter. However, in the need of very high resolution images with spatial resolution of the order of few centimeters in low cost compared to satellite installation, Unmanned Aerial Vehicle (UAV) systems or drones has paved the way for better data acquisition. According to a survey conducted by U.S. Government Accountability Office in 2012, since 2005 more than 75 countries in all over the world have adopted drones for data acquisition from different unreachable areas (U.S. Government Accountability Office, 2012). Initially UAV was utilized for military applications. However, low cost, flexibility, high mobility and safe operation make UAVs adaptable in many civilian issues viz. car counting and detection in urban areas (Zhou et al., 2018); operation monitoring of electrical appliances (Peng and Liu, 2018); anomaly identification in archaeological sites (Cavalli et al., 2013); road and non-road regions segmentation (Li et al., 2019); health assessment of historical monuments (Bacco et al., 2020) and assessment of forest areas (Cruz et al., 2017) etc.

Use of hyperspectral images for different crop type classification was addressed in (Camps-Valls et al., 2003, Zhang et al., 2016) as a conventional process because hyperspectral images carry lots of information. (Camps-Valls et al., 2003) used Support Vector Machine (SVM) and (Zhang et al., 2016) used object oriented classification process for identifying multiple crops from the hyperspectral images of the crop lands. Zhong et al. (Zhong et al., 2020) identified six different crops along with water body and roads from hyperspectral UAV image of a land using deep Convolutional Neural Network (CNN) with conditional random field classifier.

Recently several researches are carried out to use UAV RGB images for detecting specific crops and also to monitor their health conditions using different image processing tools. Torres-Sanchez et al. (Torres-Sánchez et al., 2015) utilized object based image analysis on UAV images to detect vegetation in a specific area by calculating two vegetation indices viz. Normalized Difference Vegetation Index (NDVI) and Excess Green (ExG). Wan et al. (Wan et al., 2020) extracted spectral and structural features from RGB images of rice field during its growth period to enhance the grain yield. Li et al. (Li et al., 2018) used half-Gaussian fitting method to estimate the covering of corn crops in a farmland from its UAV image. Enciso et al. (Enciso et al., 2019) used UAV RGB images to assess the height, NDVI and area covered by three different types of tomato plants. Weed mapping was done by Stroppiana et al. (Stroppiana et al., 2018) using unsupervised clustering algorithm to detect weed and non-weed areas from UAV RGB images of farmland.

Due to development of different machine and deep learning topologies for last few years, the crop classification methods from RGB images are experiencing a paradigm shift from conventional image processing methods to supervised learning techniques. Yang et al. (Yang et al., 2017) performed rice lodging assessment of a farmland using its UAV RGB images. Authors computed single feature probability for evaluating the contribution of spectral and spatial information to the overall accuracy. Malek et al. (Malek et al., 2014) proposed automatic palm tree detection algorithm to identify haphazardly planted palm oil trees from UAV RGB images of the land. In the proposed identification process, feature extraction was done using Scale-Invariant Feature Transformation (SIFT) and classification was performed by extreme learning machine classifier. Chew et al. (Chew et al., 2020) identified three different food crops (banana, maize and legume) using transfer learning process from pre trained VGG-16 and ImageNet CNN modules. In (Rebetez et al., 2016), authors proposed a hybrid classification module by merging a neural network which takes image histograms as input and a CNN module which takes raw images as input, to recognize multiple crops from aerial image of a land captured using UAV. However, merging of two neural networks increase the training time and make the classification process complex. A multi-layer Convolutional Neural Network (CNN) architecture is proposed in (Yang et al., 2020) to study phenology of rice crop from its UAV images. Bah et al. (Bah et al., 2018) proposed fully automatic learning strategy using CNN for segregating weeds from crops present at a farming land by capturing its RGB images using UAV. Fan et al. (Fan et al., 2018) detected Tobacco plant present in a land from RGB images captured by UAV system of the concerned area using two multi-layers CNNs. Bah et al. (Bah et al., 2020) designed a new CNN architecture named CRowNet by merging classical CNN and the Hough transform for detecting crop row at cornfield from its images clicked by UAVs. Kitano et al. (Kitano et al., in press) used U-net CNN architecture for corn plant identification and counting in a farmland from its RGB images captured by UAV system. Huang et al. (Huang et al., 2020) used AlexNet, VGGNet, ResNet and GoogLeNet architectures of CNN for weed mapping in rice fields. The classification process is done for two classes (rice and weed).

It can be noticed that crop identification from UAV RGB images of a farming land is gaining interest over hyperspectral images as hyperspectral images contain a lot of redundant data and that limits the training speed for supervised learning process. Further, due to Hughes phenomena, the requirement of training samples increase (Zhang et al., 2016). However, the previous literatures mainly performed binary classification with the UAV RGB images i.e. segregation of crop region (viz. rice, weed and tobacco, or tree, planted area) from non-crop region. Multiple crop classification from UAV RGB images is still not explored much, which is very essential for precision agriculture. Further it is noted that, in the previous studies multi-label crop classifications are done by using either machine learning tools or transfer learning process or standard CNN architectures, but those suffers from following limitations:

• Crop classification using machine learning tool is a bi-level process. In the upper level, some important hand crafted features are extracted from the RGB images and at the lower level analyzing those features multi-label classification is done using any machine learning classification algorithm. Success of such strategies depend on the choice of hand crafted features, and erroneous choice of features may lead to misclassification. However, in the literatures, no set of golden features are identified for efficient crop classification process, and extraction of huge number of features using hand engineering is a cumbersome job. In this regard, CNN outperforms machine learning based classification process, as CNN has inherent capability to extract huge number of features and also to perform multi-label classification.

• Some literatures used transfer learning from pre-trained CNN modules for crop identification. However, overfitting and negative transfer problems make transfer learning process erroneous.

• CNN achieved a significant progress for crop classification but they are mainly constructed by the standard CNN architectures having several convolution-pooling layers, fully connected layer (having one or more hidden layers) and classification layer. However, standard CNN architecture performs well if the images of different classes have significant distinguishable features but UAV images of crop lands are more or less looks similar for all crops. Therefore traditional CNN architecture with multiple layers cannot extract efficient features for doing proper classification among them.

From the above discussion, it is noted that application of CNN for crop identification is beneficial over the implementation of machine learning tools as CNN has already proved its efficacy for multi-label classification in case of plant disease classification (Tetila et al., 2020), health assessment of power apparatus (Ganguly et al., 2021), area segmentation in hyperspectral satellite images (Yu et al., 2019) etc. However, application of standard CNN architectures cannot extract salient features for proper classification from high resolution UAV RGB images as the aerial view UAV images of different crop lands have high resemblance among themselves. Multiple crop identification from only their aerial view UAV image is a challenging task. Due to that researchers mainly focused to segregate crop and non-crop regions in their study but not to identify multiple types of crops. Histogram based process for RGB images can do multi-label classification but that needs complex pre-processing of the actual raw images. Therefore, there is a need of designing a new CNN module for successful identification of crops from their aerial view UAV images.

To fulfil the above mentioned research objective, this article proposes a smart multiple crop identification and classification system to distinguish different crops from their UAV RGB images. This article proposes a new CNN architecture named Conjugated Dense Convolutional Neural Network (CD-CNN) for intelligent crop identification and classification from their aerial view RGB images captured using UAV systems. The proposed CD-CNN module leverages the benefits of both dense architecture by adding lower layer features with the upper layer features for deriving new salient attributes of every crop and a new Activation Function (AF) named SL-ReLU, which prevents gradient explosion during training process. Unlike standard CNNs, the proposed CD-CNN architecture can do feature extraction more efficiently from aerial view UAV images of multiple crops, reduces the feature loss and gradient dissipation due to deepening of the network. Despite of the high similarity between the aerial views of different crop farming lands, the proposed CD-CNN architecture along with SL-ReLU AF can extract salient and discriminative features from the UAV images and results higher multi-label classification accuracy compared to state of the art machine learning tools and standard CNN architectures. To the best of the authors’ knowledge, this is the first work where an automatic deep learning tool is proposed for multiple crops identification from only their aerial view UAV images. The salient contributions of this article are as follows:

• Classification Process: A conjugated dense CNN architecture is proposed for crop identification and classification from their UAV images. The dense connection provides better feature extraction and transmission. Then with the help of concatenation operation the upper layer features are coupled with the lower layer features for data fusion and efficient characteristic mining. Unlike standard CNN, the proposed CD-CNN omits the use of fully-connected layer for classification. This reduces the computation complexity and the chance of losing features due to random weight initialization at hidden layers in the fully connected layer.

• Accuracy Enhancement: A new activation function named “SL-ReLU”, proposed in (Wang et al., 2020), is used and integrated with each convolutional layer in CD-CNN to reduce the chance of unbounded convolved output, gradient explosion and poor training process due to large input data. It has been shown through theoretical and experimental analyses that SL-ReLU AF increases the feature learning ability of the CD-CNN compared to any existing function like ReLU (Nair and Hinton, 2010) and SELU (Klambauer et al., 2017) and that enhances the classification accuracy. Further, fast convergence of the training process also proves superiority of SL-ReLU over rest two AFs.

• Data Collection: Real world RGB images of five different crop lands viz. rice, sugarcane, wheat, beans and cumbu napier grass are acquired using a Quadcopter UAV system from different locations in India. The UAV flight is planned in such a way that the raw UAV images contain only a single type of crop covering region and non-crop land. To avoid ambiguity in the training process due to non-crop regions present in the images, and lesser number of original raw images, initially a large number of small candidate crop regions are extracted from the large crop region present at the aerial view RGB UAV images using Arc GIS 10.3.1 software. Further, the small candidate crop region images are fed as input to CD-CNN.

• Demonstration and Validation: The proposed architecture is able to successfully classify five different crops with an overall accuracy of 96.2%. To establish superiority of the proposed classification system, its performance is compared with existing machine learning and multi-layer CNN architectures using the similar dataset. It is observed that in terms of overall classification accuracy, the proposed crop identification and classification system outperforms the existing state of the art strategies.

The rest of the article is organized as follows: Section 2 describes the data collection process using UAV from different locations and proposed CD-CNN with SL-ReLU activation function. Section 3 depicts the experimental outcomes and comparison with other classification methodologies. Finally, Section 4 concludes the article.