Elsevier

Signal Processing

Volume 89, Issue 4, April 2009, Pages 649-655
Signal Processing

Suppressing atmospheric turbulent motion in video through trajectory smoothing

https://doi.org/10.1016/j.sigpro.2008.10.012Get rights and content

Abstract

Dynamic geometric distortion has been observed in the video captured by optical sensors in the presence of atmospheric turbulence. The distortion is caused by gradients in refractive index in the atmosphere resulting from temperature gradients in the air. This problem has been addressed previously mainly through a reference video approach which cannot handle the cases where real motion exists. In this paper, we propose a new atmospheric turbulent motion suppression algorithm that is able to suppress the turbulent motion significantly while the natural motion of objects/camera such as panning/zooming in the video is preserved.

Introduction

Atmospheric turbulence affects imaging systems by virtue of wave propagation through a medium with nonuniform index of refraction [1]. As a consequence of atmospheric turbulence, the resolution of the image is limited. The primary physical reason for atmospheric turbulence is the refraction index fluctuations. The fluctuations in the optical path of the propagating light mainly perturb the phase of the incoming light's wavefront. This causes small neighborhoods in the image to randomly move in different directions in the frames of the video. Thus, images captured by optical sensors in the presence of atmospheric turbulence are affected by degradation of resolution (blurring) and distortion of the image geometry. Viewing such video sequences disturbs the eye of the viewer since static objects appear to waver as if in motion. Such geometric distortion makes it hard for the viewer to detect real moving objects in the field.

The restoration of atmospheric turbulence-degraded images had been studied by a lot of people. For example, Labeyrie method [2], Knox–Thompson method [3], and the triple correlation method [4] represent the classic speckle interferometry techniques. Such techniques assume that a series of short-exposure turbulence-degraded images are available. Moreover, speckle interferometry techniques require a point-shaped reference star that is nearby. The reference star is used to estimate the point-spread function (PSF) of atmospheric turbulence, then the energy spectrum of the object can be estimated. The phase of the Fourier transform of the object is estimated separately by using cross spectrum method [3] or the bispectrum method [4]. In order to successfully restore an image, the noise level must be low in the series of short-exposure turbulence-degraded images. Generally, the computation demand is high. This is partially because there are many frames to be processed. The number of frames could be thousands [5] in order to get good result. Since speckle imaging method requires reference stars, this method is not applicable in restoring surveillance video which does not have a reference object.

Adaptive optics compensates for atmospheric turbulence by using mechanical means. The wavefront sensor technology [6] has been proposed to determine the phase perturbation in each short-exposure image. From the measured phase perturbation, deconvolution can then be performed by wavefront analysis [7]. Adaptive optics systems are hardware intensive.

Hybrid imaging techniques have been studied to compensate for atmospheric turbulence. In such techniques, adaptive optics hardware is used to partially correct the turbulence effects and the recorded images are then post-processed by computer. For example, Sheppard et al. [8] presented a MAP (maximum a posteriori) algorithm for deconvolution with wave front sensing.

Previous work in turbulence-degraded video restoration has mostly focused on still images and thus has not addressed the time varying turbulence effect except in [9], [10]. The Image registration in [9] uses a hierarchically windowed phase-correlation technique. The pixel in each image in a video is registered to its corresponding point in a prototype image. The initial prototype is made by time-averaging of the video. Then it is updated by the averaging of the registered video. In [10], the adaptive control grid interpolation (CGI) is used for image registration. These methods were shown to work well for video that has no real motion. The restored video has higher resolution and the video was stabilized in the sense that the turbulence-induced distortion was suppressed. However, these methods cannot properly handle the situation in which both turbulence and real motion are present simultaneously. When true motion exists, the reference frame will be obviously motion-blurred. In case of small and fast moving objects, they can even be smoothed out in the reference frames. The real motion is changed (slowed down) in the reference video because of the averaging. Consequently, the real motion is not preserved in the restored video.

An example is illustrated in Fig. 1 to show the problem of the time-averaging reference approach in processing video that has real motion. The example is a simulated panning video clip where the camera is moving from left to right. Three frames are taken to form a reference frame. Then frame 2 is warped towards this reference frame. Distortions are very obvious in the warped frame. There is no turbulent motion in the video. However, a turbulent motion effect is artificially created by the time-averaging approach, which is originally supposed to suppress the turbulent motion. These examples highlight that the time-averaging reference approach fails when real motion exists.

In this paper, we present our work in compensating for the time-varying geometric distortion. The adaptive CGI method [10] is used to compute the motion field between frames in the video. Trajectories of the pixels are then built from the motion fields between the consecutive frames. The turbulent motion is suppressed in the trajectories. While the trajectories are built, multiple frame denoising is used to improve the signal-to-noise ratio.

The remainder of the paper is organized as follows. The motion fields computation and turbulent motion compensation is presented in Section 2. Implementations and the experimental results of our approach are reported in Section 3, followed by conclusion and discussion in Section 4.

Section snippets

Turbulent motion suppression

First, we discuss the optical flow theory and the modeling of the turbulent motion, then the trajectory smoothing is presented.

Implementation and experimental results

For evaluation purposes, it is desirable to have a source of video degraded by optical turbulence together with its ground truth so that objective performance evaluation of the results can be made. Since we do not have the ground truth of the real turbulence-degraded video, simulations are made to create test sequences with both real motion and turbulent motion.

When generating the test sequences, we use the optical transfer function derived by Hufnagel and Stanley [13]:H(u,v)=e-λ(u2+v2)5/6to

Conclusion and discussion

This paper introduces an algorithm for suppressing atmospheric turbulent motion in the degraded video frames. The geometric distortions caused by atmospheric turbulence had not been paid enough attention previously. Our work shows that the turbulent motion can be effectively suppressed by trajectories centroid smoothing. The length of the smoothing window should match the period of the turbulent motion in order to suppress the turbulent motion to a maximum extend. The estimation of the period

References (14)

  • S.D. Ford et al.

    Constrained least-squares estimation in deconvolution from wave-front sensing

    Opt. Commun.

    (1998)
  • B. Horn et al.

    Determining optical flow

    Artif. Intell.

    (1981)
  • M.C. Roggemann et al.

    Imaging Through Turbulence

    (1996)
  • A. Labeyrie

    Attainment of diffraction limited resolution in large telescopes by Fourier analysing speckle patterns in star images

    Astron. Astrophys.

    (1970)
  • K.T. Knox et al.

    Recovery of images from atmospherically degraded short-exposure photographs

    Astron. J.

    (1974)
  • A.W. Lohmann et al.

    Speckle masking in astronomy—triple correlation theory and applications

    Appl. Opt.

    (1983)
  • J.C. Fontanella et al.

    Reconstruction of turbulence-degraded images using the Knox–Thompson algorithm

    Opt. Soc. Am. J. A

    (1987)
There are more references available in the full text version of this article.

Cited by (7)

View all citing articles on Scopus
View full text