Parallax-Tolerant Video Stitching with Moving Foregrounds

Abstract

The parallax artifacts introduced due to movement of objects across different views in the overlapping area drastically degrade the video stitching quality. To alleviate such visual artifacts, this paper extend our earlier video stitching framework [1] by replacing a deep learning based object detection algorithm for parallax detection, and an optical flow estimation algorithm for parallax correction. Given a set of multi-view overlapping videos, the geometric look-up tables (G-LUT) are generated by stitching a reference frame from the multi-view input videos, which map the input video frames to the panorama domain. We propose to use a deep learning based approach to detect the moving objects in the overlapping area to identify the G-LUT control points which get affected by parallax. To compute the optimal locations of these parallax affected G-LUT control points we propose to use patch-match based optical flow (CPM-flow). The adjustment of G-LUT control points in the overlapping area may cause some unwanted geometric distortions in the non-overlapping area. Therefore, the G-LUT control points in close proximity of moving objects are also updated to ensure the smooth geometric transition between the overlapping and the non-overlapping area. Experimental results on challenging video sequences with very narrow overlapping areas ($\mathord \sim$3% to $\mathord \sim$10%) demonstrate that video stitching framework with the proposed parallax minimization scheme can significantly suppress the parallax artifacts occurring due to the moving objects. In comparison to our previous work, the computational time is reduce by $\mathord \sim$26% with the proposed scheme, while the stitching quality is also marginally improved.