Keywords

1 Introduction

The digital image stitching technology has become a hot research field nowadays. It deals with remote sensing images collected by satellites or planes as well as aerial images collected by UAV (unmanned aerial vehicles). Aerial image processing [1, 2] is the key process in image stitching technology. At present, the image registration method based on the extracted image features is the main tendency in the field of image stitching. The core of the method is to find all of the matching feature point pairs by measuring similarity between each of the two images. Therefore, how to effectively extract more distinguished features from aerial images is very important.

Although there are not many existing work try to explore the feature detection, extraction and matching in aerial images, lots of algorithms are proposed to detect and extract features from other types of natural images, such as Harris algorithm [3], Canny algorithm, SIFT algorithm [9, 10], S-SIFT algorithm [11], V-SIFT algorithm [12] and so on. Among them, the SIFT algorithm has the advantage of maintaining invariance for the cases of image scaling, translation, rotation and even affine transformation. Unfortunately, aerial images have obvious different characters and properties with other types of natural images. The scattering effect generated by the atmospheric particulate matter will easily interfere with the fidelity and contrast of aerial images, leading to the loss or the degradation of the features from aerial images.

In view of these limitations, in recent years, many haze removal models and techniques are proposed to alleviate the influences from haze. Tan [5] removes the haze by maximizing the local contrast of the restored images. Fatal [6] estimates the albedo of the scene and then infers the medium transmission. Besides, in 2009, He et al. [7, 8] put forward a new kind of prior rule called dark channel prior, which will directly evaluate the transmission information of light in the haze and then remove the haze from a single input image. This effective method of removing haze can both maximally retain the feature information of the original image and adjust the overall color brightness of the image and keep the color unchanged. Unfortunately, their methods do not consider the differences in image properties between different image regions.

In this paper, we propose a novel adaptive dehaze method to adaptively estimate the optimal patch size in different regions of aerial images and remove the image haze under this consideration. We are the first to use a series of experiments to evaluate the proposed image dehaze method on aerial images. In experiments, we collect two aerial image datasets, including the images obtained by unmanned aerial vehicle, and the satellite images from Google Earth [19]. In each dataset, by using the proposed adaptive dehaze method, we obtain better performance on feature detection, image matching, and image stitching.

2 Adaptive Dehaze Method

2.1 Image Haze Removal

The key part of the famous haze removal method proposed by He et al. is they propose a new prior dark channel prior [7, 8], to estimate the transmission directly from a hazy image. Simplify, the dark channel prior means that in most of the non-sky patches, at least one color channel has very low intensity at some pixels. In other words, the minimum intensity in such a patch should have a very low value. And due to the additive airlight, a hazy image is brighter than its haze-free version in where the transmission is low. So the dark channel of these images will have higher intensity in regions with denser haze. According to haze removal approach proposed by He et al., a haze-free image can be recovered from a widely used model:

$$\begin{aligned} \mathbf {J}(x)=\frac{\mathbf {I}(x)-\mathbf {A}}{\max (t(x),t_0)}+\mathbf {A} \end{aligned}$$
(1)

where the scene radiance \(\mathbf J (x)\) is a haze-free image, \(\mathbf I (x)\) is a haze image, \(\mathbf A \) is the global atmospheric light, t is the medium transmission describing the portion of the light that is not scattered and reaches the camera and \(t_0\) is a lower bound to restrict the transmission t(x). Then, we can estimate the transmission t(x) simply by

$$\begin{aligned} t(x)=1-\omega \min _{y\in \mathbf \Omega (x)}(\min _{c}\frac{I^c(y)}{A^c}) \end{aligned}$$
(2)

where \(I^c\) is a color channel of \(\mathbf I \), \(\omega \) is a constant parameter to keep a very small amount of haze for the distant objects on images, and \(\mathbf \Omega (x)\) is a local patch centered at x. A key parameter in haze removal algorithm is the patch size \(x_n\), which is the length of \(\mathbf \Omega (x)\) in Eq. (2). In this famous haze removal algorithm, the patch size is always fixed to be 15. In our paper, we propose a novel method to adaptively remove the haze effects in different image conditions.

2.2 Image Segmentation Using Kernel Graph Cuts

In our preliminary experiments, we already found that the effects of image haze removal are different in sky and non-sky regions. Therefore, in order to obtain the optimal patch size for the sky and non-sky regions respectively, we segment the hazy images into sky and non-sky regions using Kernel Graph Cuts. Based on Boykov’s model [13], Salah [14] proposed a fully automated kernel graph cuts model, which can divide a single image into several regions unsupervisedly. In Salah’s algorithm [14], the energy equation of the simplified kernel-induced distance segmentation model is defined as follows,

$$\begin{aligned} E(\{\mu _l\},\delta )=\sum _{l\in L}{\sum _{p\in R_l}{J_K(I_p,\mu _l)}} +\alpha \sum _{\{p,q\}\in D}{r(\delta (p),\delta (q))} \end{aligned}$$
(3)

where \(J_K\) is the kernel function, \(l\in L\) is the number of the regions to be divided, L is the total number of regions and \(\mu _l\) is the parameter of the unsupervised multi-parameter Graph cuts algorithm. \(\delta \) is the penalty for each pixel marked as foreground or background. \(l\in R_l\) is a pixel that belongs to a segmented region. \(r(\delta (p),\delta (q))\) is the smooth term in neighborhood D. The first term on the right side of the Eq. (3) is the data item, the second term is the smoothing term. \(\alpha \) is the positive coefficient, which is used to adjust the weight of the data item and the smoothing item.

Fig. 1.
figure 1

Illustration of one-dimensional discrete random walk. Initially a particle is placed at \(x_n\). The black arrows marked “left” and “right” indicate two different jumping directions.

Full size image

2.3 Adaptively Obtain Optimal Patch Size Using Random Walk

After the image segmentation, we could use brute force method to obtain the optimal patch size of the sky region and the non-sky region individually on the training images. At the optimal patch size, we believe that both sky regions and non-sky regions will be extracted more keypoints after haze removal than fixed patch size. However, using exhaustive method to find the optimal patch size is a time-consuming work. Therefore, in order to make our approach efficiently, we are supposed to obtain optimal patch size effectively. Based on previous research, random walks are stochastic processes formed by successive summation of independent, identically distributed random variables and are one of the most studied topics in probability theory [15]. So, in our method, we try to find out the optimal patch size using the random walk algorithm. More specifically, we regard the procedure of searching for optimal patch size as a one-dimensional discrete random walk problem [16].

figure a

As shown in Fig. 1, a particle on a one-dimensional lattice jumps from one site to an adjacent site at random with a probability \(p_{right}\) to the right and \(p_{left}\) to the left. The algorithm to obtain optimal patch size using discrete random walk is shown in Algorithm 1. In Algorithm 1, the particle jumps in the same direction as previous jump with the probability \(k\ (0.5< k < 1)\) and the change the jump direction with the probability \(1-k\). This probability does not create preferred direction.

3 Experiments

3.1 Experimental Setting

In this part, to validate the performance of our proposed method, experiments are carried out on two sets of hazy images. The first dataset UAV is collected by our group. This data set includes 55 images obtained by the unmanned aerial vehicles. The original resolution of these images is \(4000\times 3000\). The second dataset GE is collected from Google Earth [19], which provides us a series of satellite images from all over the world. This dataset has 100 images totally, is collected from four different cities that were seriously suffering from heavy haze. In addition, the resolution of the images is \(800\times 500\) and the overlap part between images pairs is between 20% and 50%. In Table 1, we demonstrate some basic information about these four different cities.

Table 1. Specific information about four different cities in the second dataset GE
Full size table

In each dataset, we conduct the following three experiments: keypoint detection, keypoint matching and image stitching [17]. We use the OpenCV toolbox provided by Rob Hess from Oregon State University [18]. All the experiments were carried out on an Intel(R) Core(TM) 2.6 GHz PC running under Windows 10 operating system with 16.0 GB RAM.

3.2 Experiments on GE Dataset

In the experiments on GE dataset, we keep the original resolution \(800\times 500\) of images to test the performance of keypoint detection and matching. Firstly, we test the keypoint detection performance based on SIFT detector on 100 collected images. In Table 2, we show the number of keypoints detected by SIFT detector before and after haze removal. From Table 2, it is obviously that the number of keypoints increases dramatically by 262.04% on average. In Fig. 2, we show this improvement in keypoint detection task. Apparently, more keypoints will be detected after haze removal, especially those aerial images with heavy haze.

Table 2. The average number of detected keypoints before and after haze removal
Full size table

In this part, we combine 100 images into 160 pairs. Then, we evaluate the keypoint matching performance based on SIFT descriptors in 160 image pairs. In Table 3, we provide the number of matching keypoints of SIFT descriptors in the case of before and after haze removal. From this table, the average number of matching keypoints will increase about 177.89% without RANSAC [4]. With RANSAC, this average number increases about 178.20%. In Fig. 3, we demonstrate a group of example images of the keypoint matching task.

Table 3. Average number of matched keypoints based on SIFT descriptors before and after haze removal
Full size table
Fig. 2.
figure 2

A group of example images in the keypoint detection task. (a) Before the haze removal, 35 keypoints are detected. (b) After the haze removal, 2048 keypoints are detected.

Full size image
Fig. 3.
figure 3

A group of example images in the keypoint matching task (with RANSAC). (a) Before the haze removal, 19 keypoint pairs are matched. (b) After the haze removal, 744 keypoint pairs are matched.

Full size image
Fig. 4.
figure 4

A group of example images in the image stitching task. (a) is the result before haze removal, and (b) is the result after haze removal.

Full size image

In this part, we use 25 images to do the image stitching task. And the resolution of each image is \(400\times 250\). In Fig. 4, we demonstrate the result of image stitching task on two cases. Compared with the results without haze removal, our methods based on haze removal helps us to obtain better stitching results.

3.3 Experiments on UAV Dataset

In the experiments on UAV dataset, we down-sampled the images to \(1000\times 750\) to test the performance of keypoint detection and matching on lower resolution image. In this part, we firstly evaluate the keypoint detection performance based on SIFT detector on 55 collected images. In Table 4, we provide the number of keypoints detected by SIFT detector before and after haze removal. From this table, we could find the number of detected keypoints increase about 10.25% on average. This improvement shows the haze removal is effective to reduce the bad effect of haze in the keypoint detection task.

Table 4. The average number of detected keypoints before and after haze removal
Full size table

In Fig. 5, we demonstrate a group of example images in the keypoint detection task. From the comparisons of this group, we can easily find that most of newly detected keypoints are located in the regions that are badly influenced by haze. In hazy image, the buildings in a distant view (shown in the black box) cannot be distinguished clearly. And it also has a bad influence on the keypoint detection. After we utilize the algorithm of haze removal, more keypoints could be detected in these regions.

Fig. 5.
figure 5

A group of example images in the keypoint detection task. (a) Before the haze removal, 2773 keypoints are detected. (b) After the haze removal, 3453 keypoints are detected.

Full size image
Table 5. Average number of matched keypoints based on SIFT descriptors before and after haze removal
Full size table

Based on the geographic position of the collected images, we divide the collected images into seven groups. In the image matching experiment, we could combine 55 images into 167 aerial image pairs. In each pair, the overlap part is not less than 20%. In this part, we evaluate the keypoint matching performance based on SIFT descriptors in 167 pairs. In Table 5, we provide the number of matching keypoints of SIFT descriptors in the case of before and after haze removal. From this table, we could find the average number of matched keypoints will increase about 3.80% without RANSAC method. With RANSAC, this average number increases about 5.36%. This improvement also evidences that the haze removal is effective to reduce the bad influence of haze in the task of keypoint matching.

Fig. 6.
figure 6

A group of example images in the keypoint matching task (with RANSAC). (a) Before the haze removal, 147 keypoint pairs are matched. (b) After the haze removal, 179 keypoint pairs are matched.

Full size image
Fig. 7.
figure 7

A group of example images in the image stitching task. (a) is the result before haze removal, and (b) is the result after haze removal.

Full size image

In Fig. 6, we demonstrate a group of example images of the keypoint matching task. Similar with our previous results in the keypoint detection task, after the haze removal, we could obtain more matched keypoints, especially for the buildings in a distant review. These results evidence the effectiveness of the algorithms for haze removal in hazy aerial images.

Figure 7 shows the results of image stitching task on one sample image. We use four images to do image stitching task, the resolution of each image is \(200\times 150\). Compared with the results without haze removal, our method based on haze removal helps us to obtain better mosaicking results.

3.4 Adaptive Haze Removal

In this section, we apply our proposed method to obtain optimal patch size for both sky region and non-sky region individually. And we will evaluate the effectiveness of our method as well. Firstly, we divide UAV dataset into two subsets which are named as UAV-Training and UAV-Testing respectively. The UAV-Training subset, as a training set, contains 35 images. And UAV-Testing subset, as a testing set, contains 20 images. Secondly, to obtain the optimal patch size, we conduct the training on the images of UAV-Training using Algorithm 1 described in Sect. 2.3. Thirdly, to evaluate the effectiveness of our proposed method, we test the learnt model on UAV-Testing. Besides, all images in UAV-Training and UAV-Testing are down-sampled to \(1000\times 750\).

We use our adaptive method to obtain the optimal patch size for sky and non-sky region. In image segmentation, to segment a single image into two consecutive regions, the parameter L in Eq. (3) is set to 2 and the parameter \(\alpha \) in Eq. (3) is set to 10. In random walk method, the step size \(\varDelta x\) is 2, the maximum patch size N is 100, and the iteration times T is set to be 100. Besides, the value of probability k is set to 0.9. In Fig. 8, we could obtain the optimal patch size for sky region is 34 and the optimal patch size for non-sky region 40. These results obtained by our adaptive method are also consistent with the results of the exhaustive method.

Fig. 8.
figure 8

(a) The obtained candidate optimal patch sizes \(x_n\) and their corresponding average numbers \(Y_n\) of detected keypoints on the non-sky region. (b) The obtained candidate optimal patch sizes \(x_n\) and their corresponding average numbers \(Y_n\) of detected keypoints on the sky region.

Full size image

In this part, we test our adaptive model on UAV-Testing subset as well as GE dataset. The optimal patch size obtained by our adaptive model, for non-sky region is 40 and the optimal patch size for sky region is 34. As we known, in He’s method, the patch size is fixed to be 15 [7, 8] for all images or regions. In Table 6, we provide the results of keypoint detection and matching (with RANSAC) for different methods, including: the method without haze removal (without haze removal), the proposed method based on fixed patch size (fixed method), and the proposed adaptive method (adaptive method). Based on these results, we could find our adaptive method performs better than others.

Table 6. Comparisons among methods of without haze removal, fixed method and adaptive method.
Full size table

4 Conclusion

Images of outdoor scenes are usually degraded by the haze in the atmosphere. This kind of degraded images loses the contrast and color, which is possible to have a bad effect of detecting, describing, and matching image local features, or some related image applications. In this paper, we try to explore the performance based on haze removal technique using dark channel prior for aerial image obtained by the unmanned aerial vehicle. In our paper, we propose a novel method to adaptively remove the haze effects in different image conditions. We did experiments on two self-constructed datasets based on scale-invariant feature transform for keypoints detection, keypoints matching, and image stitching. Our results evidence that the haze removal is effective to decrease the bad effect of haze in these tasks. The adaptive method also demonstrates better performance than fixed one. Future work will be explored from two aspects. The first direction is to test the performance of other local feature detectors and descriptors. The second direction is to propose new real-time haze removal models for aerial images obtained by the unmanned aerial vehicles.