Multi-exposure photomontage with hand-held cameras

Highlights

• Our method abandons the requirement of full registration before fusion.

• Our method relaxes the condition of inputs and is suitable for more situations.

• Our method finds good seams to hide misalignments and handle dynamic objects.

• Our method selects proper constraints to improve final performance.

• Our method offers the prospect of more extensive applications of image fusion.

Abstract

The paper studies the image fusion from multiple images taken by hand-held cameras with different exposures. Existing methods often generate unsatisfactory results, such as blurring/ghosting artifacts due to the problematic handling of camera motions, dynamic contents, and inappropriately fusion of local regions (e.g., over or under exposed). In addition, they often require a high-quality image registration, which is hard to achieve in scenarios with large depth variations and dynamic textures, and is also time-consuming. In this paper, we propose to enable a rough registration by a single homography and combine the inputs seamlessly to hide any possible misalignment. Specifically, the method first uses a Markov Random Field (MRF) energy for the labeling of all pixels, which assigns different labels to different aligned input images. During the labeling, it chooses well-exposed regions and skips moving objects at the same time. Then, the proposed method combines a Laplacian image according to the labels and constructs the fusion result by solving the Poisson equation. Furthermore, it adds some internal constraints when solving the Poisson equation for balancing and improving fusion results. We present various challenging examples, including static/dynamic, indoor/outdoor and daytime/nighttime scenes, to demonstrate the effectiveness and practicability of the proposed method.

Introduction

High-dynamic-range (HDR) imaging techniques have been increasingly used in consumer electronics, road traffic monitoring, and other industrial, security, or military applications (Darmont, 2012). However, digital cameras often fail to capture the irradiance range that visible to human eyes. It is thus quite significant to explore effective HDR synthesis methods or detailed low dynamic range (LDR) synthesis methods. HDR synthesis methods focus on generating HDR images directly and their results are always tone mapped to LDR images which preserve details better than any of its single exposure counterpart (Debevec and Malik, 1997, Reinhard et al., 2010, Sen et al., 2012, Kalantari, 2017, Wu et al., 2018, Yan et al., 2019). Detailed LDR synthesis methods directly synthesize the result from multi-exposure images (Burt, 1984, Burt and Kolczynski, 1993, Mertens et al., 2007, Wang et al., 2018, Ma et al., 2019). Our method belongs to the detailed LDR synthesis category.

Although the multi-exposure fusion (MEF) approaches have been studied extensively, there are still some drawbacks. For instance, many existing methods have employed some kinds of merging techniques, which assume that multiple exposure images are accurately aligned (Li and Kang, 2012, Li et al., 2013, Paul et al., 2016). Thus, any misalignment due to either camera motions or dynamic contents will lead to the so-called ghosting/blurring artifacts. In the meantime, a Laplacian pyramid reconstruction scheme for image fusion was proposed in Burt and Adelson (1983), which has been widely adopted in many subsequent works (Burt and Kolczynski, 1993, Mertens et al., 2007, Shen et al., 2014). However, the method (Mertens et al., 2007) also requires the inputs to be strictly aligned. For each pixel location, every aligned candidate pixel in the stack contributes to the final pixel value. Thus, if there are any misaligned regions, the fused results would suffer from the ghosting or blurring artifacts. Fig. 1 shows two examples, where the input images are aligned before fusion, but the scenes contain dynamic textures or objects (tree leaves in the left example and moving persons in the right example). The fused results by Mertens et al. (2007) suffer from the blurry (Fig. 1 left) and the ghosting (Fig. 1 right).

Later on, some de-ghosting methods are proposed to handle aforementioned problems (Tursun et al., 2015). Firstly, some methods based on energy optimization are introduced to maintain image consistency or distinguish different parameters (Jinno and Okuda, 2008, Granados et al., 2013). Secondly, some flow-based methods realize registration with pixel-level accuracy and are effective for aligning moving objects between two images (Kang et al., 2003, Zimmer et al., 2011, Kalantari, 2017). Thirdly, some patch-based methods (Sen et al., 2012, Hu et al., 2013) are proposed to reconstruct the input images by patch-based synthesis according to one selected reference image, to form a fully registered image stack. The full alignment means that the reconstruction compensates both the camera and the scene motions. The synthesized candidates are then sent to the fusion framework. However, the patch-based reconstruction is not always robust in complicated situations, especially when encountered with dynamic textures (e.g., fountains, waterfalls, tree leaves in the wind), or structured regions. Fig. 2 shows such an example, where two patch-based methods generate blurry results in tree crown regions.,

High quality fully registration is challenging. For one thing, it is difficult to achieve a high-quality alignment under different appearances (Cui et al., 2017). For another, large foreground (Zhang et al., 2016) and near-range objects (Liu et al., 2016) would complicate the alignments, and scenes with large depth variations cannot be registered by a single homography, or by more sophisticated models (Lin et al., 2017). Besides, non-parametric approaches such as optical flows tend to generate errors at discontinuous depth boundaries (Kalantari, 2017) and the patch based reconstruction is also prone to produce errors as shown in Fig. 2.

To pursue a robust solution, the proposed method abandons the requirement of full alignment and replaces it by a rough registration with a single homography. As such, the photomontage idea proposed by Agarwala et al. (2004) is applied to compose the multi-exposure images that have been aligned roughly. However, our setting is different from Agarwala et al. (2004) in two aspects. First, Agarwala et al. generate composites interactively, which combines parts of a set of photographs into a single composite picture. Users select preferred image regions (e.g., a region containing a smiling face) at different pictures. In contrast, our solution is fully automatic because we combine image parts according to their exposure qualities. Second, the combined photos of Agarwala et al. (2004) were captured by a static tripod, whereas our inputs are captured by hand-held cameras. In our implementation, the method does not require the perfect registration, as long as it finds good seams to hide the misalignment.

The proposed method consists of some specific components as follows. It selects sub-image regions from different roughly aligned exposures by an MRF labeling and combines them seamlessly in the gradient domain. In this way, each pixel value belongs to a single image such that it is possible to maintain details well and handle blurring effectively. Moreover, it considers the dynamic identification and exposure selection in the MRF optimization simultaneously. The selected regions are not only well-exposed but are free from the interferences of dynamic objects/textures. Overall, the main contributions are:

(1) The proposed method relaxes the conditions of inputs. Conventional image alignment algorithms always fail to align inputs from hand-held cameras with large shaking. The proposed method abandons the requirement of full registrations, which can handle various complicated inputs and generate high-quality fusion results.

(2) The proposed method introduces the dynamic exclusion technique to handle moving objects. An energy optimization is first applied to detect moving objects and then a mask is generated to identify the dynamic pixels of each input, which reflects the probabilities of pixels being static or dynamic. The final results are free from ghosting with proper exposure.

(3) We propose to add some internal constraints to lighten under-exposed regions.

(4) We conduct comprehensive comparisons to demonstrate the effectiveness of our method, including objective assessment, visual comparison, complexity comparison and subjective evaluation.