Elsevier

Neurocomputing

Volume 371, 2 January 2020, Pages 124-136
Neurocomputing

Densely pyramidal residual network for UAV-based railway images dehazing

https://doi.org/10.1016/j.neucom.2019.06.076Get rights and content

Abstract

On purpose of aiding detection and recognition for railway infrastructure and dramatic changes in the environment around railways, visual inspection based on unmanned aerial vehicle (UAV) images is a highlight. However, UAV images often suffer from degradation for fog or haze, which limits the inspection efficiency. Most existing methods depend on a suboptimal two-step network with much more redundant procedures where transmission map and atmospheric light are estimated at first, and then haze-free images can be acquired using a dehazing model. This paper presents a novel end-to-end network for UAV-based railway images dehazing, and focuses on two key issues: network architecture and loss function. With regards to the first aspect, based on a pyramidal network structure, densely pyramidal residual network (DPRnet) consists of dense residual block and enhanced residual blocks, which heavily exploits the feature maps of all preceding layers and considerably increased depth at different scale, respectively. With regards to the second, a new loss function introducing structural similarity index is proposed to preserve more structural information, thereby restore the appealing perceptual quality of the hazy images. Finally, quantitative and qualitative evaluations illustrate that the DPRnet achieves better performance over the classic methods, yet remains efficient and convenient.

Introduction

Railway plays a crucial role in the development of economic and industry growth, yet the failures of railway facilities (the fastener, track, pylons and barriers) and dramatic changes in the environment around railways (the vegetation, terrain, crossing bridges and buildings) can cause potentially catastrophic accidents [6]. Unmanned Aerial Vehicles (UAVs) have become a research hotspot for infrastructure change inspection in railway fields due to its high detection efficiency and cost-benefit [55]. However, these images captured by UAVs often suffer from degradation because of suspended atmospheric particles such as fog and haze in bad weather, which directly affects the quality of photographs, thereby leads to low recognition and inspection efficiency for railway infrastructure (in Fig. 1). Image hazing approaches can effectively solve this problem by estimating a haze-free one from a hazy UAV image. Therefore, haze removal not only plays a significant role as preprocessing of UAV-based images for visual clarity effect, but also is of desirable and essential to UAV-based images for recognizing and inspecting potential dangers around railways effectively.

In the past two decades, numbers of dehazing approaches have been developed for images taken in hazy or foggy scenes. Early research work mainly focus on using multiple images and atmospheric cues to estimate depth information for haze removal are proposed, such as [32], [35]. In Ref. [34], a physical-based method is presented to locate depth discontinuities and to compute structure of a scene, from two images capturing same scene under various weather conditions. As a matter of fact, depth information is difficult to be estimated [9]. Starting from these early research works, a great deal of methods have been proposed to explicitly improve visibility, which can be classified as following categories: methods using multiple images [2], polarization-based methods [42], [43], depth-known-based methods [12], [18], prior-based method [8], [15], [20], [50], [63] and learning-based method [4], [11], [23], [24], [37], [41].

In addition, there are several dehazing methods which focus on improving the contrast and brightness of images for visual effect enhancement, thereby realize a purpose of restoration and clarity such as [30], [57], [59]. For example, Tan employs the local contrast method for image dehazing [48]. For lacking consideration of the haze-quality reduction process, nevertheless, such approaches are not applicable to different positions and images, particularly the images with more differences in scene depth [24].

For haze removal issue, we mainly present several traditional dehazing approaches including depth-known-based, prior-based methods and learning-based in Section 2, and summarize several existential problems.

In dehazing processing of previous research work [23], [24], [60], some intermediate products such as transmission map or atmospheric light are produced, which may lead to poor performance because of cumbersome process and computational complexity. In this paper, a pyramidal end-to-end network without intermediate products estimate for UAV-based railway images dehazing is presented. The structure of densely pyramidal residual network (DPRnet) consists of a high resolution (HR) network without pooling layers and a low resolution (LR) network including 2 pooling layers. For making fully use of the feature maps and optimizing the whole network, dense residual block (DRB) and enhanced residual blocks (ERB) is proposed (Note: The HR network and LR network consists of DRBs and ERBs, respectively.). The DPRnet is trained on the indoor universal dataset from China Multimedia2018 (ChinaMM2018) [1] and synthesized outdoor UAV-based dataset via a new loss function introducing structural similarity index (SSIM). Finally, the trained DPRnet can take a hazy image as input, and output a haze-free image.

This paper presents a new learning-based method for UAV-based railway images dehazing, and makes contributions as following:

  • A novel end-to-end network named densely pyramidal residual network (DPRnet) is proposed in this paper. This is achieved by embedding directly transmission t(x) and atmospheric light α into a formulation (Eq. (4)), thereby restore the haze-free image in one shot without estimating intermediate products. This also addresses the second challenge because the DPRnet does not need to estimate depth information for image dehazing.

  • A new network architecture extracting fully image features is presented in this paper, which consists of dense residual block (DRB) and Enhanced residual blocks (ERB). On the one hand, ERB can ease the network training and address degradation problem caused by increased depth. On the other hand, DRB improve information flow and keep more structural information between concatenation layers. Therefore, the proposed structure meets a large improvement of the haze removal performance.

  • An optimized loss function introducing SSIM is proposed to minimize loss between the dehazing image and corresponding ground truth one. This is more suitable for assessing perceptual image quality than other assessment methods such as MAE (Mean Absolute Deviation) in order to two factors: (1) SSIM is proposed based on the supposition that human visual perception is highly adapted to extract structural information from a scene [53], is direct method to compare the structures of the reference and the distorted images; (2) the measure for structural information modification can provide a good approximation to perceived image distortion (haze) [54].

  • The visual and quantitative comparison experiments are conducted on various datasets including indoor universal dataset (Test-I), outdoor UAV-based dataset (Test-II) and real-world dataset (Test-III). Experiments showed the proposed method achieves superior performance over the classic traditional methods.

The rest sections of the paper are showed as follows: the related work is showed in Section 2. The DPRnet includes DRBs, ERBs and loss function are elaborated in Section 3. Experimental results are presented in Section 4. The conclusion is described at the end of this paper.

Section snippets

Related work

Inspection applications based on UAV images: The UAV-based inspection scheme has become appealing for change inspection in small-scale regions for its efficient and cost-effective [45]. Recently, UAV-based aerial photography has been widely employed for crop measurements [36], engineering surveying and mapping [44], classification or recognition of multiple objects [7], [38] and detection applications such as fault detection in photovoltaic cells [5], wind turbine blade surface inspection [52],

Methodology

According to our previous investigation [55], and experiment, the hazy removal issue based on UAV images on railway scene is confronted with several challenges because of following characteristics:

  • 1)

    Varied atmospheric light on different types of samples and noise corruption [55]. For partial occlusion of surrounding railway infrastructures (catenary etc.), and shake of the UAV in a running task and other environmental factors (haze), atmospheric light in UAV images is uneven and low. Furthermore,

Experimental results

In the section, the various comparative experiments (Visual experiments, Quantitative experiments: PSNR and SSIM evaluation) are conducted on indoor universal dataset Test-I, the synthetic hazy UAV dataset Test-II as well as the real-world dataset Test-III for demonstrating the effectiveness of this proposed method (the datasets details are introduced in Section 4.2). In addition, we introduce experimental equipment, evaluation standard, datasets, training details and so on.

Discussion

Based on all above experiments and investigation, our discussion is as follows: Prior-based methods can achieve haze removal effectively without tedious parameter adjustment and training process, however they often tend to produce some sharp edges, highly contrasting colors, and is computationally intensive such as NLP [8]. DCP [15] can calculate the transmission matrix more reliably by discovering the dark channel prior (DCP), however the prior is unreliable when the atmospheric light are

Conclusions

This paper presents The DPRnet, which restore haze-free images based on an end-to-end CNN “more directly”. A new network architecture extracting fully image features is proposed in this paper, which consists of DRB and ERB. Firstly, ERB can ease the network training and address degradation problem caused by increased depth. Secondly, DRB improve information flow and keep more structural information between concatenation layers. Also, an optimized loss function introducing (SSIM) is employed to

CRediT authorship contribution statement

Yunpeng Wu: Methodology, Investigation, Software, Writing - original draft. Qin Yong: Funding acquisition, Project administration, Supervision, Conceptualization. Wang Zhipeng: Conceptualization, Data curation, Writing - review & editing. Ma Xiaoping: Software, Resources, Visualization, Validation. Cao Zhiwei: Formal analysis.

Declaration of Competing Interest

The authors declare no conflict of interest.

Funding

This research is supported by National Natural Science Foundation of China (No. 91738301) and the National Key R&D Program of China (No. 2016YFB1200203).

Acknowledgments

This research is also supported by National Engineering Laboratory for System Safety and Operation Assurance of Urban Rail Transit.

Yunpeng Wu received the M.S. degree from Southwest Forest University, China, in 2015. He is currently pursuing the Ph.D. degree with the Beijing Jiaotong University, China. His research interest includes UAV-based images processing, railway safety assurance, computer vision, and deep learning.

References (63)

  • H.C. Baykara

    Real-time detection, tracking and classification of multiple moving objects in UAV videos

  • D. Berman et al.

    Non-local image dehazing

  • CaiB.

    DehazeNet: an end-to-end system for single image haze removal

    IEEE Trans. Image Process.

    (2016)
  • ChenX. et al.

    A denoising algorithm of image for UAV based on wavelet transform and mean-value filtering

    Fire Control Command Control

    (2011)
  • L.T. Goncalves

    DeepDive: an end-to-end dehazing method using deep learning

  • N. Hautière et al.

    Towards fog-free in-vehicle vision systems through contrast restoration

  • HeK.

    Spatial pyramid pooling in deep convolutional networks for visual recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2015)
  • HeK.

    Deep residual learning for image recognition

  • HeK. et al.

    Single image haze removal using dark channel prior

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2011)
  • HuangG.

    Densely connected convolutional networks

  • M. Kaamin

    Visual inspection of historical buildings using micro UAV

  • J. Kopf
    (2008)
  • A. Krizhevsky et al.

    ImageNet classification with deep convolutional neural networks

  • LaiY.-.H.

    Single-image dehazing via optimal transmission map under scene priors

    IEEE Trans. Circuits Syst. Video Technol.

    (2015)
  • C. Ledig

    Photo-realistic single image super-resolution using a generative adversarial network

  • LiB.

    AOD-Net: all-in-one dehazing network

  • LiC.

    A cascaded convolutional neural network for single image dehazing

    IEEE Access

    (2018)
  • LiJ. et al.

    Image dehazing using residual-based deep CNN

    IEEE Access

    (2018)
  • LiR.

    Single image dehazing via conditional generative adversarial network

  • LimB.

    Enhanced deep residual networks for single image super-resolution

  • LiuC.

    A safe flight approach of the UAV in the electrical line inspection

    Int. J. Emerging Electr. Power Syst.

    (2015)
  • Cited by (29)

    • UAV imagery based potential safety hazard evaluation for high-speed railroad using Real-time instance segmentation

      2023, Advanced Engineering Informatics
      Citation Excerpt :

      Recently, most deep learning-based visual tasks, such as image dehazing [55], defect detection [4], saliency detection [56], and instance segmentation [54] utilize the deep connected network ResNet as the backbone for high detection accuracy for solving memory and time optimization issues. Nevertheless, due to the direct flow gradient between the previous layer and the later layer in ResNet, the intrinsic feature information of input images may be discarded [55], thereby leading to the low detection efficiency of instance segmentation. Fortunately, Res2Net [57], as sketched in Fig. 4 (b), is recently proposed to improve the representation of the multi-scale structure.

    • Facial image inpainting using attention-based multi-level generative network

      2021, Neurocomputing
      Citation Excerpt :

      It uses pyramid pooling to capture contextual information. Wu et al. [34] utilized the pyramid network for UAV image dehazing to get global prior information. Different from it, we utilize convolution with stride 2 to keep structures and details by integrating feature information.

    • Attentive U-recurrent encoder-decoder network for image dehazing

      2021, Neurocomputing
      Citation Excerpt :

      Multi-scale information is helpful to obtain global information and recover details. Hence, current CNN-based methods usually incorporate multi-scale information into the framework, e.g. DRPN [6], MSCNN [13] and EPDN [25]. In this section, we further compare our method with these 3 CNN-based methods.

    View all citing articles on Scopus

    Yunpeng Wu received the M.S. degree from Southwest Forest University, China, in 2015. He is currently pursuing the Ph.D. degree with the Beijing Jiaotong University, China. His research interest includes UAV-based images processing, railway safety assurance, computer vision, and deep learning.

    Yong Qin received the Ph.D. degree from China Academy of Railway Sciences in 1999. He is currently a professor in the State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, China. His research interests include deep learning, transportation information engineering and railway safety assurance.

    Zhipeng Wang received the B.S. and Ph.D. degrees from Northeast University and Beihang University, China, in 2008 and 2014. He is currently an associate professor in the State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, China. His research interests include machine learning and fault diagnosis.

    Xiaoping Ma received the B.S. and Ph.D. degrees from Beijing Jiaotong University, China, in 2009 and 2018. He is currently a post-doctoral in the State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, China. His research interests include machine learning and fault diagnosis.

    Zhiwei Cao received the B.S. degree from China University of Mining and Technology, China, in 2017. He is currently pursuing the Ph.D. degree with Beijing Jiaotong University, China. His research interest includes traffic safety, computer vision, and image processing.

    View full text