Edge-guided Composition Network for Image Stitching

Highlights

• We propose a novel deep learning framework for composition in image stitching.

• We illustrate the importance of structure consistency preserving in image composition, and leverage the structure prior provided by a proposed perceptual edge branch to further enhance the composition performance.

• We build a Real Image Stitching Dataset (RISD), which is the first general-purpose benchmark for image stitching training.

Abstract

Panorama creation is still challenging in consumer-level photography because of varying conditions of image capturing. A long-standing problem is the presence of artifacts caused by structure inconsistent image transitions. Since it is difficult to achieve perfect alignment in unconstrained shooting environment especially with parallax and object movements, image composition becomes a crucial step to produce artifact-free stitching results. Current energy-based seam-cutting image composition approaches are limited by the hand-crafted features, which are not discriminative and adaptive enough to robustly create structure consistent image transitions. In this paper, we present the first end-to-end deep learning framework named Edge Guided Composition Network (EGCNet) for the composition stage in image stitching. We cast the whole composition stage as an image blending problem, and aims to regress the blending weights to seamlessly produce the stitched image. To better preserve the structure consistency, we exploit perceptual edges to guide the network with additional geometric prior. Specifically, we introduce a perceptual edge branch to integrate edge features into the model and propose two edge-aware losses for edge guidance. Meanwhile, we gathered a general-purpose dataset for image stitching training and evaluation (namely, RISD). Extensive experiments demonstrate that our EGCNet produces plausible results with less running time, and outperforms traditional methods especially under the circumstances of parallax and object motions.

Introduction

Image stitching plays an important role in many applications like remote image mosaicking [1], panoramic videos [2], virtual reality [3], and panorama creation software. Conventionally, image stitching is a process of combining multiple images with overlapped areas to produce a wide-view panorama [4]. There are two ways of mitigating the artifacts on the stitched image: (1)improving alignment by increasing the available degrees of freedom [5], [6], [7], [8], [9]; (2) hiding misalignments by image composition techniques such as selecting better seams in the overlapped areas. However, artifacts caused by large parallax and objects moving can hardly be concealed by better alignment, thus improving the image composition is the only remedy in these cases.

The most widely used composition technique for image stitching is seam-cutting [10], [11], [12], [13] and blending [14], [15]. The former aims at finding an invisible seam in the overlapped region of the aligned images and the latter is to smooth the transitions between images near the seam. Seam-cutting can be viewed as a $\{0,1\}$ labeling problem that creates a mapping between pixels in the composite and source images. The optimal seam is found by minimizing an energy function that describes the differences between images in the overlapped region. Many efforts have been devoted to improving seam-cutting performance by penalizing the photometric difference using various energy functions. The euclidean-metric color difference and gradient difference [12], [16], [17] are the earliest attempt. Later, other features like texture complexity [18], canny edges [19], [20], and saliency [21] were also taken into consideration. However, the performance of these methods is limited by the weak hand-crafted features. Because of the difficulty to find the best combination and balance the weight of these features, traditional methods often lack of adaptively to various input conditions. Moreover, these hand-crafted features only gather very local information and lack of stronger geometric information guidance, thus may produce structure inconsistency results, such as tearing or cropping recognizable objects (Figure 1). In terms of computation time, the energy-based composition methods all have high time complexity by solving with min-cut or dynamic programming, which is far from being a real-time application. Therefore, it is highly desirable to have a more efficient image composition method that is robust to varying input conditions and capable of utilizing features adaptively.

Recently, convolutional neural networks (CNNs) has greatly promoted the development of computer vision. With the help of CNNs, more discriminative and adaptive image features can be extracted, and more robust performance could be reached with the data-driven method. To the best of our knowledge, the only learning-based work related in image/video stitching is by Lai et al. [22], which learns a smooth spatial interpolation between input videos. Different from theirs, we focus on the image composition process and cast the whole process as a blending problem. To seamlessly blend the pre-registered images into an artifacts-free stitched image, we leverage CNNs to develop an Edge-guided Composition Network (EGCNet) to regress the blending weights. The insight is that the pre-registered images should be blended at regions with high similarity of them to avoid inconsistent and noticeable image transitions. To learn this composition nature better, we use Siamese network [23] for features extraction, and combine the feature maps not only through concatenation but also a group-wise correlation layer to explicitly provide similarity measurement, so the following layers can predict blending weights from the combined features. Our work can be seen as continuing the second line of image stitching research.

Training such a network to predict generic composition for image stitching requires a sufficiently large training set. However, the existing image stitching datasets are all too small for effective training. Thus, to provide sufficient training data, we build a Real Image Stitching Dataset (RISD) of 500 pairs of images with various parallax levels and scene types. Each sample of RISD contains a pre-registered image pair and a visually plausible composite image as ground-truth. With the help of RISD, our EGCNet can be trained in an end-to-end manner. Meanwhile, since preserving structure consistency is of great significance in the composition process, we explicitly exploit the perceptual edges as guidance. To achieve this, we build a two-branch network with a composition branch and a perceptual edge branch. The perceptual edge branch detects edges of the input pre-registered images and guides the composition branch in two aspects. On the one hand, edge features generated in the perceptual edge branch are embedded into the composite branch to provide structure prior. On the other hand, edge maps produced by the perceptual edge branch are utilized in our proposed edge-aware smoothness loss and detail-preserved loss, which guide the model to pay more attention to regions with strong structural information. The whole model is trained in a joint way of supervision and self-supervision, which is supervised by the ground-truth composite images in RISD and self-supervised by the edge-aware losses that were designed based on the task characteristics.

In summary, our main contributions are as following:

• We propose a novel end-to-end deep learning framework for composition in image stitching.

• We illustrate the importance of structure consistency preserving in image composition, and leverage the structure prior provided by a proposed perceptual edge branch to further enhance the composition performance.

• We build a Real Image Stitching Dataset (RISD), which is the first general-purpose benchmark for image stitching training.