Robust online rain removal for surveillance videos with dynamic rains

Abstract

The current rain removal techniques proposed for surveillance videos mainly assume consistent rains with invariant extents and types, and are implemented in a batch-mode learning manner. Such assumption deviates from the continuously varying insights of practical rains, and the batch mode further makes the techniques infeasible for real long-lasting videos. To alleviate these issues, this study proposes a novel online rain removal approach to represent practical dynamic rains embedded in surveillance videos. Particularly, we model the rain streaks scattered in each video frame as a patch-wise mixture of Gaussians (P-MoG) distribution, and update its parameters frame by frame. Such a P-MoG modeling manner finely reflects the non-i.i.d. dynamic variations of rains along time. In specific, the P-MoG rain model in each frame is regularized by the learned rain knowledge in previous frames, making the online model adaptable to not-identically-distributed rains in each frame while regularized by not-independently-distributed rains in previous frames. The proposed model is formulated as a concise probabilistic MAP model, which can be readily solved by EM algorithm. We further embed an affine transformation operator into the proposed model, making it adaptable to a wider range of videos with camera jitters. The superiority of the proposed method is substantiated by extensive experiments implemented on synthetic and real videos containing static and dynamic rains as compared with state-of-the-arts in both accuracy and efficiency.

Introduction

Nowadays, tremendous surveillance cameras have been installed almost all over the world, and large amount of surveillance videos have been and are being collected automatically, for facilitating various subsequent video processing tasks, like interest in object detection [1], [2], traffic congestion [3], action recognition [4], [5] and so on. However, such collected surveillance videos are always with unsatisfactory qualities due to complicated environmental variations, conducting difficulties of further processing missions. One of the most typical videos with such issues are those containing rain streaks. Actually, videos captured in the wild inevitably contain rainy frames where the rain extent always dynamically changes and consistently lasts for a long time. The appearance of rain streaks in a video form a layer of bright streaks adhered to the clean background, and always causes severe quality corruption and blurry artifacts to the videos. Designing effective rain removal techniques to enhance the quality of surveillance videos has thus become a critical issue and been attracting much research attention recently [6].

Many approaches have been presented for this rain streak removal issue on surveillance videos. The early attempts have tried to establish analytic models to extract rain components from videos by discovering specific and distinctive attributes for representing rains. Typical utilized attributes include chromatic [7], [8], photometric [9], [10], [11], and specific temporal/spatiotemporal [12], [13] properties of rains, as well as those existed in frequency domain [8], [14]. In the recent decade, there is a new trend for handling this task, by directly designing a MAP model for decomposing a rainy video into a rain layer and non-rain ones [12], [15], [16], [17]. The key for this modeling task is to extract appropriate prior expression for rain streaks contained in the video for representing insightful features underlying this specific subject. An elegant design of prior always leads to good performance of a rain removal method. Most frequently used priors along this research line include low rank [12], [15], [16], [17], similarity across non-local area [11], [12], [13], [18] and so on.

However, there are still significant limitations existed in the current rain removal methods for surveillance videos collected from practical scenarios. One typical issue is that most current methods either employ one or more fixed regularization terms, with pre-specified parameters, for representing rains [7], [11], [13], or assume rains i.i.d. distributed across frames imposed with local structures [19]. Such a modeling manner is suitable for encoding rainy videos with relatively consistent extent of rains throughout time. Nevertheless, in practically obtained videos, the rain types as well as its extents are always highly diverse and dynamically changed across time, due to both inevitable weather change and illumination condition variation. Especially, in practical long-time rainy videos, rain streaks over frames are always demonstrated as a highly dynamic and non-i.i.d. distributed configurations. On the one hand, the rains are always not identically distributed and evidently different along time. In some frames the rains might be pouring and the rain streaks are heavily scattered across the frame, while in others, the rains might be light and some tiny rain drops are weakly distributed over the frame. On the other hand, the distribution shapes of rain streaks over one frame are generally closely related with its adjacent frames across the video, and thus such rain distribution is not independent in time. Most of the current methods for video rain removal have not specifically considered such complicated characteristic of dynamic rain streak variations, especially its temporal non-i.i.d. distribution essence across a video sequence, which makes them always not able to robustly adapt practically long-term videos containing rains with dynamic changes in time.

There is another critical issue to realize an online approach to handle the rain removal task for a long-lasting surveillance video containing rains. While the surveillance videos are continuously collected in practice, most of current methods are designed on an entire video sequence and implemented in a batch-mode learning manner. This makes them only suitable on a limited length of videos, while cannot efficiently fit the consistently incoming video frames lasting for a long time in practice. It is thus also significant to rebuild efficient methods for such an rain removal issue for constructing rational online algorithms to make the task able to be efficiently implemented on practical consecutive surveillance videos.

To alleviate the aforementioned issues, this study presents a novel online method for the task. The main idea is to formulatethe distribution of rain streaks as a patch-wise mixture of Gaussians (P-MoG), whose parameters vary from frame to frame, and dynamically and associatively updated along time. The utilization of P-MoG for encoding rain knowledge is inspired by our previous work [17], which can finely represent the local structure pattern underlying rain streak distributions. As compared with most previous methods, this new method has some specific characteristics: Firstly, it employs an online mode to incrementally implement rain removal in a frame-by-frame manner along a video sequence. Compared with conventional batch-mode manner, such an online paradigm is more efficient, and more available on long-lasting videos. Secondly, for each frame, the method learns a specific P-MoG distribution with distinctive parameters, which can well fit the specific configuration of rains located on the frame to be differentiated from others. This finely encodes the “not identical” property of rain streak distributions along time. Thirdly, when tuning parameters of rain distribution in each frame, the previously learned knowledge on such distribution can regularize them by a KL-divergence term, to make those parameters not too much deviated from previous ones, and enforce relationship between rain knowledge of this frame and those learned from previous ones. This finely reflects the “not independently distributed” property of rains temporally. Through designing such novel learning regime, our method can be robustly and efficiently calculated on surveillance videos with dynamic rains.

In summary, this paper mainly makes following contributions for rain removal tasks on surveillance videos:

• An online rain removal method is designed for the task, which can be efficiently implemented in long-lasting surveillance videos. To the best we know, this is the first online approach proposed against this task.

• A concise probabilistic model is designed for reflecting the dynamic rains contained in videos. The model contains a likelihood term, functioning to specifically fit rain shapes in each new-coming frame, and a KL-divergence regularization term, enforcing the learned rain knowledge not too far away from the previously learned ones. In such a modeling manner, the practically non-i.i.d. rain distribution can be finely delivered. To our best knowledge, this is the first method considering dynamic variations of rain extents/shapes in videos for the task.

• An entire MAP model is constructed by fully encoding rains as aforementioned, and integratively formulating the backgrounds and moving objects contained in a surveillance video. A transform operator is further embedded into the model to make it adaptable to videos with camera jitters. An EM algorithm is readily designed for solving the model, each step capable of being efficiently solved in closed-form. Experiments substantiate the advantage of the method on the task in both time and accuracy.

The paper is organized as follows: Section 2 provides relevant research works on video/image rain removal. Section 3 presents the proposed online rain removal model and its solving algorithm. Section 4 further extends the method to videos with camera jitters. Experimental results are shown in Section 5, and conclusions are finally made in Section 6.