Delineation of cultivated land parcels based on deep convolutional networks and geographical thematic scene division of remotely sensed images
Introduction
Cultivated land parcels (CLPs) are the basic units on which humans engage in agricultural activities using land resources (Sun et al., 2020). Accurate information of CLPs is the basis and core information required for digitalization and informatization in modern agriculture (García-Pedrero et al., 2017, García-Pedrero et al., 2018). The acquisition of information on the spatial morphology of CLPs is a prerequisite for extracting other information (e.g., use types or cropping systems of CLPs) (Wu et al., 2019). Therefore, it is important to extract accurate spatial morphological information of CLPs (i.e., CLPs boundary information) which has vital value for practical applications (Masoud et al., 2020).
With the construction of the Global Navigation Satellite System (GNSS), it has become possible to obtain CLPs boundary information by using equipment or sensors placed in the fields (Chen et al., 2021, Sarri et al., 2017). Although the method using GNSS is very accurate, it requires a large amount of positioning equipment. Thus, this is expensive and time-consuming for large area agriculture monitoring. As a technologically feasible way, remote sensing provides a comprehensive, timely, and rapid record of various geographical phenomena, patterns, and processes on the earth's surface. Thanks to the development of sensor technology, the spatial resolution of remote sensing images has gradually increased, reaching the meter level and even the sub-meter level. It is common to use high spatial resolution remote sensing images as data sources to obtain boundary information of CLPs in a lot of research (Valero et al., 2016). Apart from the traditional manual visual interpretation, automatic information extraction technology with computers has become a popular way to outline CLPs boundaries (Waldner and Diakogiannis, 2020, Zhang et al., 2020). The state-of-the-art of CLPs boundaries delineation with remote sensing images is as follows.
- •
Boundaries delineation of CLPs using edge based image segmentation (edge detection methods)
Early edge detection methods mainly refer to differential operators, like Roberts (Roberts, 1963), Prewitt (Prewitt, 1970), Sobel (Sobel, 1978), Canny (Canny, 1986), and so on. These operators are theoretically capable of detecting all kinds of edges from the image. With the improvement of algorithms, other methods (e.g., wavelet transform, Hough line detection, and etc.) can also detect image edges in different ways. For boundaries delineation of CLPs, Wagner (2020) used the Sobel operator to detect edges from Sentinel-2 imagery (Wagner and Oppelt, 2020b). Robb (2020) used Canny and phase congruency edge detection algorithms to delineate edges from unmanned aerial system imagery (Robb et al., 2020). Although the algorithms are simple and easy to be implemented, the methods are relatively sensitive to noise, and there are some discontinuous lines in the extracted results. In other words, the methods do not guarantee closed polygons (North et al., 2019). Thus, some post-processing methods need to be utilized after edge detection. For example, Wagner (2020) proposed an improved parametric activity contour model and a graph-based contour line extraction method to extract parcel polygons (Wagner and Oppelt, 2020b), and Robb (2020) used Hough line detection and active contour model to derive CLPs boundaries based on edges obtained by Canny (Robb et al., 2020).
- •
Boundaries delineation of CLPs using region based image segmentation (region segmentation methods)
This method can be subdivided into two methods: region growth and region split-merge. Region growth segmentation is a bottom-up approach. Multiresolution segmentation (MRS) based on fractal net evolution approach and mean shift (MS) segmentation (Ma et al., 2014, Ming et al., 2012, Ming et al., 2015) are representative methods for region growth segmentation. Specifically, MRS has been integrated in eCognition software, which is famous for object based image analysis (OBIA). In contrast, region split-merge segmentation operates images from top to down. For CLPs boundary information extraction, O’Connell (2015) extracted parcels by MRS using airborne remote sensing images with two strategies (i.e., single scale and multiscale) (O’Connell et al., 2015). Ming (2016) delineated CLPs using scale pre-estimation based MS segmentation from SPOT-5 images (Ming et al., 2016). These methods can relieve sensitivity to noise to a certain extent, but they are complex compared to edge detection methods. Region based segmentation methods are usually unsupervised, the segmentation results also need post-processing, such as image classification, to select the CLPs. In addition, the choice of segmentation parameters (especially the scale parameter) is one of the important issues that cannot be avoided (Ma et al., 2015, Ming et al., 2018).
The former two methods have their own strengths and weaknesses. Thus, researchers want to combine the advantages of both edge-based and region-based approaches. For example, Mueller (2004) pointed out that the traditional region-based image segmentation method has the problem that the boundaries of adjacent parcels are mixed and cannot be distinguished (Mueller et al., 2004). Therefore, shape information, as a kind of edge feature, is involved into the region-based segmentation method, so that the boundary of CLPs could be extracted accurately.
- •
Boundaries delineation of CLPs based on image classification with deep learning methods
Considering the disadvantage of image segmentation methods, binary classification (i.e., boundaries of CLPs and non-boundaries) is involved in the extraction of CLPs boundaries as well. Thanks to deep learning networks, image deep features can be extracted. Image classification with deep learning methods perform better than traditional statistical classifiers in a lot of remote sensing applications (Ma et al., 2019). For delineation of CLPs boundaries, Wagner (2020) extracted a variety of shallow features, such as spectra and textures of images, and used them as input of the neural network for CLPs boundaries delineation (Wagner and Oppelt, 2020a). Taravat (2021) used improved U-Net to extract the boundaries of CLPs (Taravat et al., 2021). In addition to convolutional neural networks (CNN) based on the patch and semantic segmentation networks, some networks converting the traditional hard classification into soft classification have been proposed to extract edge or boundary features of images specially, for example, HED (Xie and Tu, 2017), DeepEdge (Bertasius et al., 2015), RCF (Liu et al., 2019, Liu et al., 2017), CASENet (Yu et al., 2017), CEDN (Yang et al., 2016), and etc. Thus, the outcomes of these networks are usually edge or boundary probability maps, which the ground objects with clear boundaries, such as roads, buildings, coastal lines, and so on, can be easily extracted. Liu (2020) employed RCF to delineate farmland parcels with high spatial resolution remote sensing optical images and time series synthetic aperture radar (SAR) data, which worked well in mountainous areas (Liu et al., 2020). Though deep learning methods require plenty of labelled images for model training, they are seen as potential approaches to delineating CLPs because of the deep abstract features they used.
Though great achievements have been made, a common problem still exists that whatever kind of method is used in the previous studies mentioned above, the experimental data they used only covered limited areas and contained very few types of ground objects. However, CLPs information with large areas and complex landscape conditions is urgently desired in agricultural practical applications. Thus, these large covered images need to be processed before information extraction. One of the necessary processing between traditional image pre-processing and information extraction is image region division (Xu et al., 2019, Zhou et al., 2018). According to the first law of geography (Tobler, 1970) and the rule of territorial differentiation, it can be concluded that there is a clustering effect (spatial homogeneity within the region) for the same type of ground objects, and there is regional heterogeneity for different types of ground objects. Therefore, region division, an application of classical geographic idea, is increasingly becoming an effective solution for solving the problem oriented to large areas and complex landscapes.
In recent years, researchers have also studied the issue of image region division to varying degrees in the remote sensing community. In this article, the methods of image region division are categorized according to different division indicators.
- •
Image region division by auxiliary features
The division indicators of this method are always from auxiliary data, which associate with the study area. The auxiliary data include vector data (Wu et al., 2020), digital elevation model (DEM) (Wu et al., 2021), or other remote sensing images (Huang et al., 2021). For example, DEM was used to divide the study area into mountain regions, hilly regions, plain regions, and so on (Liu et al., 2020). Then, information extraction was followed in different subregions. Zhou (2019) employed point cloud data acquired by light detection and ranging (LiDAR) technique to generate the standard normalized digital surface model which was further used as a kind of auxiliary feature to divide images into different regions (Zhou et al., 2019). In the urban research field, image region division was always used to identify the basic units of urban functional zones. For example, Zhang (2017) used vector data of city roads to derive different districts (Zhang et al., 2017). Huang (2021) utilized nighttime light and daytime multi-view imagery to obtain the image regions (Huang et al., 2021). However, this kind of method by auxiliary features highly requires auxiliary data which is sometimes unavailable. In addition, transferring the scale or registering the auxiliary data to the target remote sensing images is often required.
- •
Image region division by image shallow features
The fundamental shallow features of remote sensing images are spectral features and texture features. These two kinds of features are always used as the basis for image segmentation and classification. Similarly, they are also used as indicators for image region division. For example, Zheng (2013) carried out over-segmentation with the spectral feature of images (Zheng et al., 2013) and then used improved Markov random field model to combine the over-segments as post-processing (Zheng et al., 2019, Zheng et al., 2017). Considering OBIA and the idea of stratified processing, Xu (2019) segmented the image on rough scale based on the fused spectral and texture features to achieve image region division. After that the information of farmland could be derived from the rural region (Xu et al., 2019). In addition to spectral and texture features, there are also some other specific indicators retrieved from remote sensing images for image region division. Some indicators also have a clear physical meaning, such as surface temperature, vegetation indices (e.g., normalized difference vegetation index, NDVI), and so on. Contrasted with the method based on auxiliary features, the method using image shallow features can relieve the limitation of data dependency. However, the results of this method are binary classification commonly. It is not good at generalizing image information on a large scale because of the broken regions obtained by this method. Additionally, it is needed to indicate that image segmentation is a necessary process whatever for method based on auxiliary features or based on shallow features, thus scale selection issues are often inevitably involved in image region division by using these two kinds of methods above.
- •
Image region division by deep features of images
Semantic segmentation networks by deep learning are the core of this method. In computer vision, semantic segmentation is a classic task. Semantic segmentation is a way to classify the image pixel by pixel, which is different from traditional segmentation in the remote sensing community. At an early stage, semantic segmentation based on graph theory (for example, grab cut, N-cut, and etc.) merely classified the image into two categories, and semantic information was not clear. With the reemergence of deep learning and CNN, the development of semantic segmentation has been greatly promoted. Long (2015) proposed fully convolutional network (FCN) by deleting fully connected layers of traditional CNN (Long et al., 2015, Shelhamer et al., 2017), which has made great progress on semantic segmentation. After that, researchers proposed other kinds of semantic segmentation networks based on FCN, including SegNet (Badrinarayanan et al., 2017), U-Net (Ronneberger et al., 2015), DeepLab (Chen et al., 2018), and so on. These networks extract deep features of images on multiscale and are capable of generalizing the image on a large scale by continuously stacking convolutional layers and pooling layers. The deep learning networks can yield more satisfactory image semantic segmentation results compared with the graph-based methods.
Inspired by the big progress of semantic segmentation in computer vision, deep learning based semantic segmentation has been introduced into the remote sensing community and have made some achievements in the field of image processing and analysis (Ma et al., 2019). Semantic segmentation is currently used for two tasks, namely, binary classification (thematic objects extraction) and full objects classification. Although image region division belongs to the latter, there is few studies on image region division. This is because the existing semantic segmentation datasets are mostly for thematic objects extraction and image classification tasks. There is a lack of remote sensing image datasets specifically for image region division, resulting in limitation of the related research to a certain extent. However, the method of image region division using semantic segmentation networks is theoretically feasible and advantageous.
In conclusion, image region division is a necessary image processing and gradually becomes an effective method to solve the problem of information extraction from remote sensing images with large areas and complex landscape conditions. Therefore, aiming at fine extraction of CLPs, this article proposed a stratified extraction framework with two steps. Firstly, image region division using a semantic segmentation network on a coarse scale, and then, delineation of CLPs boundaries using a boundary detection deep learning network on a fine scale. Experiments proves that the proposed framework can improve the accuracy and efficiency in CLPs boundaries extraction.
The remaining parts of this article are settled as follows. The description of the proposed methods is introduced in Section 2. Section 3 shows the experiments and results in detail. The effectiveness of image region division and the performance of the proposed methods are discussed in Section 4. Finally, the conclusion of this article is summarized in Section 5.
Section snippets
Methodology
The flowchart of CLPs boundaries delineation in this article is shown in Fig. 1. First of all, this article proposed a novel theory of image region division based on geographical thematic scene division according to the rule of territorial differentiation. The definition of geographical thematic scenes and their division system are described in Section 2.1. Secondly, the dataset of geographical thematic scene division and the dataset of CLPs boundaries are built by using pre-processed remote
Study area and datasets
In this article, two study areas have been chosen to carry out experiments. Fig. 7 shows the true color synthesis of the study areas. Study area A is located in Guangping County, Hebei Province, China. The experimental data is a Gaofen-2 fusion image with the panchromatic band and the multispectral image. The image was photographed on 25 February 2017, with the size of 6000 × 6000 pixels and a spatial resolution of 1 m, namely, 36 square kilometers. The main type of crop in study area A is
Effectiveness of image region division based on geographical thematic scenes
Focusing on practical agriculture monitoring in a large area, fast CLPs boundaries delineation is the fundamental step. As the expansion of the study area, the types of objects contained in the study area increase and get complex besides CLPs. It is difficult to accurately delineate CLPs information within the large area image covered by complex landscapes. Therefore, according to the first law of geography and the rule of territorial differentiation, this article proposes a novel method to
Conclusion
The purpose of this article is to delineate CLPs boundaries from high spatial remote sensing images by using deep convolutional networks. The proposed method is capable of extracting CLPs within large area covered by complex landscapes, which can meet the need of practical applications. The main contributions are summarized below.
(1). Image region division gradually become a potential solution for information extraction of large area remote sensing images with complex landscapes. According to
Funding
This work was supported in part by the National Natural Science of China under Grant 41671369, in part by the National Key Research and Development Program under Grant 2017YFB0503600-05, in part by 2021 Graduate Innovation Fund Project of China University of Geosciences, Beijing, under Grant ZD2021YC054, and in part by the Fundamental Research Funds for the Central Universities.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (60)
- et al.
Field-road trajectory segmentation for agricultural machinery based on direction distribution
Comput. Electron. Agric.
(2021) - et al.
Detection of cropland field parcels from Landsat imagery
Remote Sens. Environ.
(2017) - et al.
Urban functional zone mapping by integrating high spatial resolution nighttime light and daytime multi-view imagery
ISPRS J. Photogramm. Remote Sens.
(2021) - et al.
Training set size, scale, and features in Geographic Object-Based Image Analysis of very high resolution unmanned aerial vehicle imagery
ISPRS J. Photogramm. Remote Sens.
(2015) - et al.
Deep learning in remote sensing applications: a meta-analysis and review
ISPRS J. Photogramm. Remote Sens.
(2019) - et al.
Scale parameter selection by spatial statistics for GeOBIA: Using mean-shift based multi-scale segmentation as an example
ISPRS J. Photogramm. Remote Sens.
(2015) - et al.
Cropland extraction based on OBIA and adaptive scale pre-estimation
Photogramm. Eng. Remote Sens.
(2016) - et al.
Edge- and region-based segmentation technique for the extraction of large, man-made objects in high-resolution satellite imagery
Pattern Recogn.
(2004) - et al.
Wide-area mapping of small-scale features in agricultural landscapes using airborne remote sensing
ISPRS J. Photogramm. Remote Sens.
(2015) - et al.
Delineation of agricultural fields in smallholder farms from satellite images using fully convolutional networks and combinatorial grouping
Remote Sens. Environ.
(2019)