Keywords

1 Introduction

Remote Sensing (RS) has become an important system to obtain a huge amount of data, especially in precision agriculture processes. The association between information technology and agricultural procedures has been useful for obtaining and processing farm data, resulting in tools for productivity estimation, nutritional evaluation, water management, pests and diseases detection. Among several devices, multispectral and hyperspectral sensors attached to airplanes or Unmanned Aerial Vehicles (UAVs) can obtain images with detailed spectral information that helps identifying and distinguishing among materials spectrally similar [1].

Properties extracted from reflectances in some ranges of the electromagnetic spectrum can be better evaluated by arithmetical combinations of different spectral bands [2]. These combinations usually use ranges from visible to Near Infrared (NIR) frequencies and are measures of vegetation activities. These measures are called Vegetation Indices (VIs) [3].

Especially for Brazilian agriculture, costs for acquiring data represent one of the most important barrier to the improvements provided by multispectral and hyperspectral sensors. Both sensors and their respective analytical platform are high-cost systems and they can not be offered as a set-off for familiar and/or small producers. For instance, the price of a hyperspectral camera is above tens of thousands of pounds. Furthermore, vehicles that transport these sensors (Drones and UAVs) are susceptible to mechanical and also human failures, leading to crashes during flight. Besides damaging the drone, the high cost spectral cameras could be damaged too. Common Red-Green-Blue (RGB) cameras are currently low cost sensors with potential to estimate some kind of VIs from visible spectral bands, such as Modified Photochemical Reflectance Index (MPRI) [4], that is applied to light use efficiency and water-stress. Nevertheless, the VI most used and accepted by agronomists is the Normalized Difference Vegetation Index (NDVI). This index uses NIR band and R band from RGB.

There are some papers that describe methods to obtain NIR images from ordinary cameras. Hardware alterations on cameras are described in [5, 6], removing NIR blocking filters from them. In [5], a new CFA (Color Filter Array) was developed to obtain a color image and a NIR image at the same time. A method to obtain NDVI images directly from a common camera is proposed in [6]. The first step to obtain this kind of images was removing the NIR blocking filter from one of the RGB channels of the camera and adding a low pass filter, allowing to obtain only NIR information. In this work, B channel from RGB was replaced for NIR. In 2016, a method to estimate NIR images from RGB images was proposed [7]. RGB images were captured by a camera attached to an UAV on different days and hours as well as NIR images obtained by a special camera, but these images were not from an aerial vehicle. After a regression analysis using RGB and NIR images, experiments showed that G channel from RGB is highly correlated to NIR, thus, they concluded that it is possible to estimate NIR images from G channel.

This paper introduces a method for estimating near infrared spectral information from RGB images, using R, G and B values and material endmembers. The purpose is making viable development of tools based on cheaper RGB cameras capable of estimating accurately NIR bands for VIs and other agricultural applications, making UAVs technology more attractive and accessible to familiar and small producers.

The rest of the text refers to the following: Sect. 2 – describing the hyperspectral image data used in the development, and the experiments for endmembers extraction; Sect. 3 – presenting the proposed method; Sect. 4 – describing the experimental results; and Sect. 5 – about conclusions and future works.

2 Spectral Data and Endmembers Extraction

2.1 Hyperspectral Images

Hyperspectral images were used as source of spectra to create a database of endmembers [8] for this experiment. Pure spectral signatures plays an important role on classification of materials on hyperspectral images. Because of the lack of a huge free hyperspectral image dataset, some well known images from literature were used to collect spectral information. These images have a ground truth image, allowing to assign each pixel spectral signature to different kinds of ground cover like vegetation, bare soil, minerals and water. Three public available hyperspectral scenes indicated in the literature were chosen to make spectral analysis: Indian Pines, Salinas and Pavia Centre. Information about these hyperspectral images can be obtained in [9]. RGB images were obtained using the three equivalent wavebands from the original hyperspectral images.

In addition to literature images, an image mosaic from a citrus cultivation in Lampa, Santiago, Chile, was used to do spectral analysis too. It was captured on January 17, 2011 by using a HySpex VNIR-1600 sensor attached to an airplane. The image was acquired between 12:49 and 12:54 p.m, with altitude of 2,500 m above sea level. Each image pixel corresponds to 0.5 m of spatial resolution, with 160 spectral bands, ranging from visible wavelengths to NIR (411.2 nm–988.9 nm). To generate a RGB image for this image (Fig. 1), bands 55 (611 nm), 41 (560.2 nm) and 12 (453.8 nm) were chosen to obtain respectively R, G and B channels.

Fig. 1.
figure 1

RGB image from citrus cultivation in Lampa, Chile.

Full size image

2.2 Endmembers Extraction Experiments

Pixels from selected images were taken to represent classes from their respective ground truths, resulting in 400 pixels per class, but for Indian Pines image was possible to get only 15 pixels per class, because some classes have less than 50 pixels in its ground truth. These selected pixels were used to find spectras that best represent each class (endmember). Automatic Target Generation Process (ATGP) [10], Pixel Purity Index (PPI) [11], N-FINDR [12] and Fast Iterative Pixel Purity Index (FIPPI) [13] were chosen to find these endmembers and verify the best algorithm to use on proposed method.

In addition to endmembers extraction results, an average of spectra was assessed for each class. An example of mean spectral signature can be seen at Fig. 2.

Fig. 2.
figure 2

Example of mean spectral signature. (a) 400 spectral signatures. (b) Mean spectral values from (a).

Full size image

For the Lampa image, sixteen class labels were categorized for representing areas of soils (different conditions), lakes and crops. Each class from this image has an endmember associated with it. The same procedure is done to all images selected.

A total of 12,848 pixels were sampled to create a training database. In order to validate the method, 4,190 pixels spectra were sampled from other image areas. In Table 1 is shown the class label with their respective number of pixels.

Table 1. Training database samples distribution.
Full size table

3 Proposed Method

Figure 3 shows a block diagram that summarizes the proposed method. At the top, it is seen the hyperspectral image block, since the whole development is based on pixels sampled from this type of images. Then at the right side it is shown a block of endmembers extraction. These endmembers are hyperspectral pixels chosen to represent classes, as described in the previous section. At the left side it is seen the RGB image block, which represents the RGB pixel values extracted from the hyperspectral image data; and at the middle it is seen the ground truth pixels representing the classes. Using the RGB image input and the ground truth, the KNN classification is applied, to obtain the spectral signatures estimate of the RGB image pixels. It is important to explain that this block diagram is referred to the development diagram, since after this development, the NIR band estimation is based on RGB image and ground truth, without the use of the hyperspectral image data. The method will be described in detail in the following paragraphs, starting from KNN classification, because Hyperspectral images, RGB images acquisition, ground truth and Endmembers extraction have already been described in the previous section.

Fig. 3.
figure 3

Block diagram representing the proposed method.

Full size image

The WEKA tool [14] was used for the experiments. The K-Nearest Neighbours (KNN) algorithm was chosen as the instance classifier. Input attributes were R, G, B and MPRI values.

MPRI is a VI based on normalized difference between two spectral bands in visible wavelength, specifically, red and green [5]. The MPRI equation is expressed as following:

$$ MPRI = \frac{{\left( {R_{Green} - R_{Red} } \right)}}{{\left( {R_{Green} + R_{Red} } \right)}} $$
(1)

where \( R_{Green} \) is the green reflectance value and \( R_{Red} \) is the red reflectance value.

WEKA KNN returns a vector with a belonging degree to each class for the classified instance (pixel). At first, the algorithm creates an n dimension array called dist, which n is the number of classes from classification problem. Each dist element has an initial value, called classifier correction, defined by Eq. (2), where N is the number of instances.

$$ correction = \frac{1}{{MAX\left( {1,N} \right)}} $$
(2)

For each k nearest neighbors the algorithm calculates a weight W using the distance between them and the sample to be classified, using the number of attributes (\( x \)) from input data, Eq. 3.

$$ W_{i} = \frac{\sqrt x }{{dist_{i}^{2} }}, \;1 \le i \le k $$
(3)

The dist array is updated on positions corresponding to each of k nearest neighbors, as can be seen on Eq. (4). After updating dist, the array is normalized, generating the proximity degree, or probability distribution Pi, Eq. (5).

$$ dist_{i} = correction + W_{i} $$
(4)
$$ P_{i} = Norm_{i} = \frac{{dist_{i} }}{{\mathop \sum \nolimits_{j = 1}^{k} W_{j} }}, \;1 \le i \le k $$
(5)

Pi is used to calculate a weighted sum that origins new spectral signature, such as:

$$ S{^{\prime}} = \mathop \sum \nolimits_{i = 1}^{{n_{c} }} P_{i} \overline{S}_{i} $$
(6)

where \( n_{c} \) is the number of class labels, \( P_{i} \) is the probability of pixel belong to class i and \( \overline{S}_{i} \) is the endmember array of class i. This new spectral signature calculated by Eq. (6) is based on the principle that hyperspectral pixels are composed by a mixture of endmembers from different targets [8]. Figure 4 illustrates how this step of obtaining new spectral signatures works, so that, at the left side it is shown the k = 3 nearest neighbors, represented by their endmember signatures, and their respective proximity degree, Pi, from the RGB pixel; and at the right side, the resulting estimative of spectral signature for the RGB pixel. Using this estimated spectral signature for all the pixels of the RGB image, it is possible to estimate its NIR band.

Fig. 4.
figure 4

Obtention of new spectral signatures from weighted sum using endmembers and KNN proximity degree.

Full size image

In order to evaluate this experiment results, Root Relative Squared Error (RRSE) was calculated between classes’ endmembers and validation data, wavelength by wavelength, using their respective KNN classification label. RRSE is given by:

$$ RRSE(j) = \sqrt {\frac{{\mathop \sum \nolimits_{i = 1}^{n} \left( {\overline{S}_{j,i} - V_{j,i} } \right)^{2} }}{{\mathop \sum \nolimits_{i = 1}^{n} \left( {\overline{V}_{j} - V_{j,i} } \right)^{2} }}} $$
(7)

where \( n \) is the number of samples from validation dataset, \( \overline{S}_{j,i} \) refers to endmember reflectance for wavelength j, \( V_{j,i} \) is the validation dataset reflectance value for wavelength j. \( \overline{V}_{j} \) is a mean value of reflectance values from validation dataset pixels for wavelength j.

4 Experimental Results

Endmembers were extracted with ATGP, PPI, N-FINDR, FIPPI and also mean spectral signature. Image segmentation with Spectral Angle Mapper (SAM) [15] and Spectral Information Divergence (SID) [16], spectral similarity measures, were used to verify which algorithm could be used to feed the estimation method with the best endmembers. SAM and SID algorithms use endmembers as a class reference pattern to classify spectral signatures of pixels from hyperspectral images, analyzing how far or near these pixels are from endmembers. Table 2 shows accuracy of segmentation performed after endmembers being extracted by these methods.

Table 2. Image segmentation accuracy (%) with SAM and SID.
Full size table

According to Table 2, Mean Spectral signature outperformed classical algorithms of endmembers extraction, presenting the best segmentation results using SAM and SID classifiers. For the literature methods of endmembers extraction, FIPPI spectral signatures presented best results in segmentation task. Therefore, Mean Spectral signatures and spectral signatures selected by FIPPI were used on NIR estimative experiments.

The best performance on training datasets indicated 5 neighbors and weighted distance inverse (Euclidean distance) for KNN. Table 3 shows KNN classification accuracy using RGB and MPRI data from data set (Sect. 2.2) as input. As can be seen, Salinas image data showed best accuracy result, so it was chosen to make NIR estimative and also Lampa image, to verify estimative results on an image that doesn’t have ground truth defined by remote sensing specialists.

Table 3. KNN classification accuracy to the data sets.
Full size table

The KNN classification accuracy to the Lampa image was 64.46%. After KNN classification, the RRSE was calculated between each sample spectra from validation data and the mean spectral signature which has the same label given by classifier. RRSE mean value between validation and average spectra is 58.88% to the Salinas data set and 52.86% to the Lampa data set (blue curve on Fig. 5). For experiments done with spectral signatures extracted with FIPPI, RRSE mean value between validation data and FIPPI endmembers is 67.68% to the Salinas data set and 70.22% to the Lampa data set (blue curve on Fig. 6).

Fig. 5.
figure 5

RRSE between real validation data and average spectral signature (blue); and RRSE between new spectral signature and average spectral signature (green) (Lampa Image). (Color figure online)

Full size image
Fig. 6.
figure 6

RRSE between real validation data and spectral signatures extracted with FIPPI (blue); and RRSE between new spectral signature and endmembers extracted with FIPPI (Lampa image). (Color figure online)

Full size image

Estimated spectral signatures were obtained by applying Eq. 6 for each sample of the validation dataset and RRSE was calculated for the two selected images using Mean Spectral Signatures (green curve on Fig. 5) and FIPPI spectras (green curve on Fig. 6). In average, RRSE for all wavelengths was 8.12% to the Salinas estimated image and 8.48% to the Lampa estimated image. RRSE results calculated between estimated spectras and FIPPI endmembers were 9.04% to the Salinas data and 10.14% for Lampa data in average. It is possible to see how RRSE get lower error values per band for estimated spectral signatures in comparison to the RRSE calculated in relation to the spectral signature from hyperspectral images for both Mean Spectral Signatures and FIPPI endmembers. Although new spectral signatures being a kind of endmembers mixture, they are still closer to the endmembers than spectral signatures from validation data for each classified pixel with KNN.

Entire Salinas image was used to show the power that this method has to estimate NIR spectral information, creating full RGB and NIR images for visual comparison. Since Lampa image has high dimension, an image region (400 × 400 pixels) was extracted from Fig. 1, also for a visual comparison between real images and those estimated by the proposed method. The estimative were performed using Mean Spectral Signatures and the results to the Salinas image are shown in Fig. 7. Results of estimation to the Lampa region image are shown in Fig. 8. Note that estimated images preserve a lot of transitions (high frequencies) among image components (classes), with good similarity with original images.

Fig. 7.
figure 7

Estimated image to Salinas Data Set.

Full size image
Fig. 8.
figure 8

Estimated image to Lampa image

Full size image

At Figs. 9 and 10, NDVI pseudo color images are showed both NDVI calculated with original NIR data and NDVI calculated with estimated NIR data. NDVI values between 0.4 and 0.8 were considered to generate color ranges, because this NDVI value range shows the state of vegetation health. Red areas show uncovered soil and unhealthy vegetation. Colored areas (orange to green) shows different health state of vegetation, being green areas related to healthiest vegetation. Note that NDVI maps assessed using original NIR data and those based on NIR estimatives are very similar.

Fig. 9.
figure 9

NDVI maps for Salinas image. (Color figure online)

Full size image
Fig. 10.
figure 10

NDVI maps for Lampa image. (Color figure online)

Full size image

5 Conclusion

In this paper, a method for estimating spectral signatures from RGB images using KNN was proposed. The use of a weighted sum using belonging degrees to classes has shown high potential for estimating spectral signatures for each image pixel, with an error smaller than 9% for NIR bands using Mean Spectral Signatures as endmembers, also resulting in quite similar NDVI maps. This kind of method makes feasible the use of accessible technologies to familiar and small producers. Some applications have high correlation between visible band and other bands, such as NIR. Thus, low-cost RGB cameras can be applied for obtaining adequate estimations in agriculture. Despite its good results, the method has limitations such as dependence on having ‘pure’ spectral signatures (endmembers) to estimating NIR bands and knowledge of the flight region to create a precise ground truth for KNN classification step. Ongoing work has shown many possibilities for improvements, including local correlations, measurements with a hand-held spectroradiometer and use of images with few bands (multispectral images).