Occlusion detecting window matching scheme for optical flow estimation with discrete optimization
Introduction
In estimating optical flow between a reference image and a target image, occlusion refers to a certain region of the reference image that does not correspond to any region in the target image due to movement of objects and/or view change. Unless properly defined, occlusion degrades the quality of estimation, particularly on object boundaries, and may lead to severe performance degeneration in many applications of optical flow estimation; for example, frame interpolation [1], [2], motion segmentation [3], [4], motion layer ordering [5], and motion compensated coding [6], [7].
A convincing method to find exact occlusion is grasping exact motion of all objects in the images; inversely, if we obtain exact occlusion in advance, the accuracy of flow estimation will be significantly improved. In practice, neither the exact motion nor the exact occlusion is provided in advance, thus it is very challenging to obtain highly accurate optical flow and occlusion at the same time. We address this challenge with a novel window matching scheme on an unified discrete MRF (Markov Random Field) framework.
Various approaches have been presented for jointly estimating optical flow while detecting occlusion. Many of them employ individual sequential steps as following: (1) calculating optical flow as if occlusion does not exist, (2) finding occlusion based on the estimated flow, and then 3) iterating the previous two steps until convergence. One simple approach to detect occlusion given flow estimation, is thresholding the residual of subtracting warped target image from reference image [8]. In [9], authors introduce a probabilistic criterion employing histogram of image contents, and alternately calculate flow and visibility through the EM-algorithm. Alvarez et al. define occlusion by checking symmetric consistency of forward and backward flows [10]. Another method in [11] also detects occlusion by cross-checking the bi-directional flows, utilizing a coarse-to-fine approach with discrete optimization. To reduce inherent computational complexity, it estimates movement of similar pixel groups (i.e., super-pixel) with over-segmenting input images. A method in [12] utilizes observation that a certain point in a target image could probably be occluded if the point is accessible by multiple pixels in a reference image through forward warping. It finally refines the estimated flow with the probability map of accessibility. All those approaches, however, may suffer from the fact that they depend on the initial flow result which could be incorrectly estimated in the occluded area; and subsequent iterations may also yield erroneous results accordingly. Moreover, they require additional computational cost for obtaining the backward flow or the occlusion probability map.
Occlusion has also been recognized as a significant issue in stereo matching problems. Zitnick et al. proposes to iteratively update a 3D disparity array using uniqueness and smoothness constraints, detecting occlusion by thresholding [13]. The uniqueness constraint implies that each pixel in a target image should have at most one correspondence in a reference image. An approach in [14] shows promising results by applying the Graph-cuts algorithm [15] to efficiently enforce the uniqueness constraint. A method in [16] uses backward disparity and visibility maps to detect symmetric occlusion using iterative optimization with the Belief Propagation [17]. These methods generally find good solutions in the discrete sample spaces; however, in the two-dimensional flow estimation, the size of the sample space exponentially increases, leading to very high computational complexity unless efficiently managed.
Meanwhile, Ballester et al. presented an assumption that an occluded pixel may be visible in the previous frame of a reference frame [18]. But their approach is limited to the case that multiple frames are provided and motion across the frames is relatively simple. A recent method shown in [19] utilizes over-segmentation to find image layers with respective movements, detecting occlusion with local ordering of the layers. While this method can address large occlusion issues on textureless regions, it may yield over-simplified flow estimation depending on performance of the over-segmentation. Fortun et al. also presented an approach to manage large occlusion problems [20], [21]. They first compute local flow candidates on non-occluded regions, and then fill in a large occlusion based on the candidates close to the occlusion. However, filling-in may fail if no region is close enough or multiple confusing region candidates exist.
In [22], authors showed a new model incorporating a cost for occlusion which is supposed to be very sparse in the input images within infinitesimal time interval. While this method presents state-of-the-art performance in detecting occlusion, it degenerates performance of flow estimation as the process iterates. In addition, the performance can be very sensitive to a threshold value controlling sparseness of the occlusion. Another work [23] also applies the sparsity constraint estimating flow as well as occlusion; but the algorithm eventually depends on the weight map obtained from motion inconsistency, yielding insufficient performance to be state-of-the-art.
Section snippets
Proposed approach
This work aims to simultaneously estimate optical flow and detect occlusion within a single optimization framework. Our method does not iterate through flow estimation and occlusion detection. Compared to the previous state-of-the-art method [22] without the iteration, the proposed approach does not degrade the performance of optical flow estimation, indeed it does not require sensitive threshold parameter tuning.
Background
Let be an undirected graph with a node set and an edge set . A node in corresponds to a pixel in a reference image. Let ls be a label, i.e., a random variable for a node s in some discrete sample space representing the quantized vector set . A vector in is three dimensional, i.e., . First two dimensions represent a displacement vector for x and y directions, homogeneously quantized by L labels in each direction. The last dimension os ∈
Window matching scheme
Data matching cost in our work is based on a square window with support-weight [26], [27], which can be defined as follows: where W(s) is a neighboring node set in the window supporting s. means the support-weight function for s in the reference, and indicates the function for s′ in the target. s and t are mapped to s′ and t′ by displacement vector of us(ls). ρ(t, t′) denotes a similarity measure between pixels at t
Optimization
To find the optimal solution for the MRF formulation in (1), we employ the TRW-S [29], which has shown state-of-the-art results [30] in many discrete framework applications. The asymptotic computational complexity of the TRW-S in general is and for our current framework, we rewrite it as . As the proposed method requires an adequate number of labels to yield satisfactory estimation results, we introduce techniques to address the complexity issue which is dominated by the
Experiments
All the experiments are performed on a system with 3.30 GHz Intel Core i5-2500 CPU and Nvidia Geforce GTX 285 GPU (240 CUDA core.) We validate our method on various image sets from the Sintel dataset [36] and the Middlebury flow dataset [37]. To assess the accuracy of estimated flow, we compare average end-point error (AEPE) and average angular error (AAE). For the performance of occlusion detection, we calculate the F1 scores using the ground-truth occlusion maps.
We assumed the maximum
Conclusion
In this work, we presented a novel support-weight based window matching method for simultaneously estimating optical flow and detecting occlusion. Our method works on a unified optimization framework, which does not require explicit flow estimation nor additional backward flow computation for occlusion detection. The proposed support-weight provides an effective clue to detect occlusion, and improve estimation of flow in the occluded area. Experiments showed our method yields highly competitive
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIP) (2015R1A5A7036384), and the grant (no. 14-2016-010) from the Seoul National University Bundang Hospital (SNUBH) Research Fund.
References (38)
- et al.
Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow
Comput. Vision Image Understanding
(2016) - et al.
Combination of spatio-temporal and transform domain for sparse occlusion estimation by optical flow
Neurocomputing
(2016) - et al.
Adaptive large window correlation for optical flow estimation with discrete optimization
Image Vis. Comput.
(2013) - et al.
Efficient mrf deformation model for non-rigid image matching
Comput. Vis. Image Understanding
(2008) - et al.
Moving gradients: a path-based method for plausible image interpolation
ACM Trans. Graphics (TOG)
(2009) - et al.
Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation
IEEE Trans. Circuits Syst. Video Technol.
(2007) - et al.
Motion competition: a variational approach to piecewise parametric motion segmentation
Int. J. Comput. Vis.
(2005) - et al.
Simultaneous motion estimation and segmentation
IEEE Trans. Image Process.
(1997) - et al.
Motion layer extraction in the presence of occlusion using graph cuts
IEEE Trans. Pattern Anal. Mach. Intell.
(2005) - et al.
Improving frame interpolation with spatial motion smoothing for pixel domain distributed video coding
5th EURASIP Conference on Speech and Image Processing, Multimedia Communications and Services
(2005)