An overview on spectral and spatial information fusion for hyperspectral image classification: Current trends and challenges
Introduction
Development of hyperspectral sensors provides hyperspectral images (HSIs) containing hundreds spectral bands. The spectral signature of each image pixel constituted by hundreds spectral bands acts as a finger print for identification of its material type. A HSI is a cube constituted of images acquired from the same scene but at different electromagnetic wavelengths where each slice of this cube is associated with a special wavelength (see Fig. 1). In other words, each pixel of HSI (spatial sample) located in row i and column j denoted as p(i, j) has a spectral signature composed of the associated reflections of that position of image scene in various wavelengths (a feature vector containing the associated values of different spectral bands). The huge spectral information simplifies distinguishing between different materials. Thus, it allows material recognition and land cover classification with a high accuracy. HSIs with rich spectral information are useful in various applications and fields such as mineralogy, agriculture, load cover classification and target detection [1].
Although the single use of spectral features may be useful but it may not be enough in many cases. When two different objects have the same spectral signatures, they can be discriminated through their shapes and texture [2]. Thus, one can proposed to fuse the spectral and spatial information to improve HSI classification. The main and based idea of using spatial information is that in local regions, neighboring pixels have similar spectral features and belong to the same class with a high probability [3]. To better understand the value of using spatial information, please attend to Fig. 2, where, the position of pixels are randomly changed, and, the spectral features of each pixel remained unchanged. The result of spectral classification is equivalent for both of these figures. But, Fig. 2(a) contains valuable spatial information about shape and texture of objects which can be used in a spectral-spatial classifier. A significant HIS classification improvement can be achieved by applying an appropriate spectral-spatial fusion method.
To extract the spatial information, usually a local window is considered around each pixel of image. By applying a spatial transform or by computing the statistics of the local dependency, some spatial features are extracted and assign to the central pixel [4].
The spectral-spatial fusion methods are generally categorized in three main groups (Segmentation based, Feature fusion based, and Decision fusion based) where each of them also contains some sub-groups. This categorization is shown in Fig. 3 and represented as follows:
- A)
Segmentation based methods
This category of the spectral-spatial fusion methods produce some segments (objects or super-pixels or Pixons) through the HSI. This technique is based on the fundamental assumption that the scene is segmented into objects such that all samples (pixels) from an object are members of the same class; hence, the scene's objects can each be represented by a single suitably chosen feature set. Typically the size and shape of objects in the scene vary randomly, and the sampling rate and therefore the pixel size are fixed. It is reasonable to assume that the sample data (pixels) from a simple object have a common characteristic. A complex scene consists of simple objects. Any scene can thus be described by classifying the objects in terms of their features and by recording the relative position and orientation of the objects in the scene.
In the segmentation methods, the spatial information is used to generate segments. Each segment contains the adjacent pixels with similar spectral features. Two approaches can be used to benefit the obtained segmentation maps:
A-1. The obtained objects are classified instead of pixels. In other words, the same label is assigned to all pixels belong to an object.
A-2. The HSI is classified pixel-wise. Then, the obtained segmentation map is used as a mask to improve the pixel-wise classification map. Usually, the majority voting rule is used to assign the same label to all pixels located in a segment.
The segmentation based methods remove the noisy pixels of the classification maps but selection of an appropriate segmentation algorithm, generation of suitable objects, with fitting sizes and shapes is a challenging task.
- A)
Feature fusion based methods
In the spectral-spatial feature fusion category, the spectral and spatial features are extracted individually or simultaneously. Then, the obtained spectral-spatial feature cube is fed to a potential classifier to achieve the classification map. Various feature fusion methods are represented as follows:
B-1. Features stacking
In these methods, the spectral features and the spatial ones are extracted individually, and then simply stacked together to generate the spectral-spatial cube. These methods are relatively simple, but due to independent extraction of spectral and spatial features procedure, the hidden information in joint spectral and spatial features will be lost. Moreover, the stacked spectral-spatial feature vector assigned to each pixel has a high dimension, which results in curse of dimensionality with a limited number of available training samples (Hughes phenomenon) [5].
B-2. Joint spectral-spatial feature extraction
Some fusion methods instead of individual extraction of spectral features and spatial ones, jointly extract them. Some of advantages of these methods are: avoiding the long fused vectors, due to features stacking, and considering joint contribution of spectral and spatial information. Of course with the cost of, more computation and missing some information of the original spectral bands.
B-3. Kernel based classifiers
The spectral and spatial features can be combined through applying multiple kernels or composite kernels. The high potential of kernels in extraction of non-linear features allows to handle the non-linear class boundaries. But, designing of an appropriate kernel and selection of its parameters is a hard task.
B-4. Representation based classifiers
The representation based methods are the non-parametric ones with no requirement to any assumption about data distribution or statistics estimation. The most well-known methods of this category are sparse representation (SR) and collaborative representation (CR). These methods are based on this idea that each image pixel can be represented through a linear combination of atoms of an appropriate dictionary. The dictionary composition (or dictionary learning), and solving the optimization problem is a difficult task.
B-5. 3D spectral-spatial feature extraction
Due to 3D inherent of HSI, simultaneously extraction of spectral and spatial features preserves the joint dependencies of spectral and spatial information. 3D filters are usually selected for extraction of 3D spectral-spatial cube. The high volume of computations, selection of appropriate 3D filters and their parameter settings are difficulties of these methods.
B-6. Deep learning based classifiers
Deep learning methods such as conventional neural networks (CNNs) extracts joint spectral-spatial features layer by layer where sub-feature map of each layer is extracted from feature map of the previous layer. The high potential of deep learning methods in extraction of non-linear and hidden features is the main advantage of them. However, the deep learning networks have hyper-parameters that need to set where learning of network requires a large training set. Otherwise, the over-fitting problem causes less classification accuracy in the testing phase compared to the training stage.
- A)
Decision fusion based methods
In the decision fusion methods, the classification map is obtained multiple times through applying different classifiers with the same feature set; or by individually applying the same classifier to various feature sets; or by applying various classifiers to various feature sets. The final classification map is obtained by implementation of a decision fusion rule such as majority voting and joint measures method. Selection of feature sets or choice of classifiers containing complement information; and high computation time due to implementation of multiple classification processes are difficulties of the decision fusion methods.
A review of different information fusion methods is given in this paper. Several state-of-the-art methods from each represented group are introduced. The advantages and disadvantage of each group are also discussed.
Section snippets
Segmentation based (object) methods
There are two types of segmentation methods. In the first type, a segmentation algorithm is applied to the HSI for objects extraction. Then, the objects are classified. In the second type, the obtained segmentation map is used as a mask for relaxation of a pixel-wise classification map. The main challenge of the object based methods is selection of an appropriate segmentation algorithm that extract a sufficient number of valid objects and avoids over-segmentation or under-segmentation [4].
A
Feature fusion
The HSI classification methods in the feature fusion level can be done in two general approaches. In the first approach, the spatial features are extracted from the HSI and then, the extracted features are combined with the spectral features through a combination method such as feature stacking or kernel based methods. In the second approach, the spectral-spatial features are extracted jointly to preserve the correlated nature of HSI cube where the spectral and spatial information is
Decision fusion method
Due to intrinsic limitation of each single feature set, the HSI classification methods by using just a single feature set ignore some valuable information and loss elegant details. To improve the classification accuracy, it is proposed that use several feature sets containing complement information to avoid information losing. In the decision fusion level, which is a high level fusion, separate decisions based on individual feature sets are drawn, and then, the results are combined to conclude
Experiments
A brief representation of advantages and disadvantages of different spectral-spatial fusion methods for hyperspectral image classification is seen in Table 1. For each subgroup of three types, a method is given as an instance. The performance of these methods are assessed in terms of classification accuracy and computation time using tree real and popular hyperspectral images: the well-known Indian Pines, University of Pavia and Salinas.
The Indian scene was collected by Airborne
Trends and advanced fusion methods
Three main trends have been seen in the recent literature:
- 1)
Design of new feature extraction methods for generation of rich spectral-spatial features with a high ability in class discrimination and preserving the 3D local and global structure of hyperspectral images.
- 2)
Hybrid fusion methods where two or more types of fusion methods are used for feature fusion, decision fusion and classification map relaxation (regularization).
- 3)
Deep learning methods for joint spectral-spatial feature generation with
Conclusion
The fusion topic in the image processing field has been discussed from two main views in literature. In the first view, the useful features from two or more individual source images are fused to provide an image with all beneficial characteristics of the source images [120], [121], [122], [123], [124]. In the second view, an image containing various worthful characteristics is explored from different aspects. The useful features are extracted and fused together to allow more powerful decision
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (142)
- et al.
Segmentation andclassification of hyperspectral images using watershed transformation
Pattern Recognit.
(2010) - et al.
Superpixel-based 3D deep neural networks for hyperspectral image classification
Pattern Recognit.
(2018) - et al.
A dual-kernel spectral-spatial classification approach for hyperspectral images based on Mahalanobis distance metric learning
Inf. Sci. (Ny)
(2018) - et al.
Segmentation and classification of hyperspectral images using watershed transformation
Pattern Recognit.
(2010) - et al.
Binary coding based feature extraction in remote sensing high dimensional data
Inf. Sci. (Ny)
(2016) - et al.
Feature space discriminant analysis for hyperspectral data feature reduction
ISPRS J. Photogramm. Remote Sens.
(2015) - et al.
Attribute profile based feature space discriminant analysis for spectral-spatial classification of hyperspectral images
Comput. Electr. Eng.
(2017) - et al.
Two dimensional linear discriminant analysis for hyperspectral data
Photogramm. Eng. Remote Sens.
(2015) - et al.
Facial expression recognition using sparse local Fisher discriminant analysis
Neurocomputing
(2016) - et al.
Fuzzy local discriminant embedding for image feature extraction
Comput. Electr. Eng.
(2015)