A fast hybrid retargeting scheme with seam context and content aware strip partition

Abstract

Image retargeting as a basic trick has been successfully applied to various computer vision problems. In this work, we propose a fast non-continuous seam carving with seam context (FN2SC) for robust and efficient image retargeting. FN2SC first conducts content aware image partition to separate an image into several strips with different target sizes from their content importance. It helps the removed seams in key objects distribute in a relatively uniform manner and prevents distortions to key objects. Then FN2SC performs fast non-continuous seam carving controlled by seam context of both neighboring relationship and touch bound relationship of seams. The seam context makes the removed seams distribute scattered and relieves artifacts caused by seam carving. During seam carving, image distortion is monitored by fast content aware image distance. Finally, FN2SC switches seam carving to scaling when the distortion meets the tolerance, which resizes the strips to the target sizes for image retargeting. Specifically, fast seam searching and image distortion based switching make FN2SC a fast and effective hybrid scheme. Experimental results demonstrate that the proposed FN2SC approach achieves good performance in terms of image quality and efficiency comprehensively.

Introduction

With the rapid growth of display device diversity, the same contents are required to display with different resolutions or aspect ratios. Image retargeting attempts to investigate an optimal solution taking both display size and human visual system into consideration. An effective image resizing scheme is necessary [1]. In addition, as Yan and Tan stated in [2], image retargeting is not just a scaling algorithm, but a thought which can be applied to various fields such as content-based image retrieval [3] for achieving low bit-rate image retrieval, aesthetics enhancement to rearrange the object [4], and video synopsis [5] etc. A good image retargeting algorithm should be effective in preserving details and preventing distortions, and be efficient simultaneously to resize an image.

Traditional scaling approaches suffered from distortions distributed throughout the entire image and deformed prominent objects distinctly. To overcome this shortcoming, many content aware approaches have been proposed. These approaches could be classified into continuous [6], [7], discrete [1], [8], hybrid [9] and semantic-based [10], [11], [12] approaches.

The continuous algorithms resize images by optimizing a content aware objective function including several types of constraints. With these constraints, the important regions would be preserved approximately to the original size compared to the unimportant regions. Wolf et al. [6] resized a video using non-homogeneous warping based on local importance. Zhang [13] employed shrink ability maps and random walk model to accelerate the scaling process and decrease the storage requirements. Wang et al. [14] presented a scale-and-stretch scheme to update a warped image that matches optimal local scaling factors iteratively. Guo et al. [15] constructed a mesh image representation which was consistent with the underlying image structures. However, the emphasis of relative scale of salient object would inevitably distort its nearby objects. Kim et al. [16] proposed a divide-and-conquer approach to media retargeting based on Fourier analysis. Zhang et al. [17] presented a shape-preserving approach to ensure that the new shapes of prominent objects are geometrically similar to their original shapes both locally and globally. But the method cannot preserve edges of key objects in retargeting. Huang et al. [18] presented image retargeting with preserving the global structure in images, whose accuracy relied on robust structure detection methods. Wu et al. [19] developed symmetry-summarization to catch and summarize repetitive structural contents in an image with the case of structure overlapping. Dong et al. [20] proposed a framework to retarget the textural regions with example-based synthesis and deal with non-textural regions by fast multiple operators. In the continuous algorithms, the neighboring pixel values changed smoothly and usually there existed no apparent artifacts. They were often optimized globally. However with continuous algorithms, contents in the resized images were obtained by interpolation which makes the details of key objects distorted to some degree in the resized images.

One of the typical discrete approaches is seam carving [1], [8], [21]. Seam carving (SC) [1] searched and removed the pixels causing the least loss of energy. Because the removed pixels mostly distributed in the unimportant regions, the removing would cause small image distortions to the key objects and the remaining pixels were preserved the same with those in the original image. Therefore, the resized images kept much details. However, seam carving takes two main disadvantages: (1) Two neighboring pixels in the resized image are possibly far away from each other in the original image, which takes a high risk of resulting in scene discontinuity and artifacts. (2) It is time-consuming to scan the entire image and remove a single seam. Rubinstein [8] modified the criterion of removing a seam from the minimum energy to the minimum energy change. Cho [22] proposed importance diffusion to constrain the seam distribution and reduce the artifacts. These approaches outperformed the traditional seam carving [1] in terms of artifact reduction. Grundmann et al. [23] proposed non-continuous seam carving with discontinuous seams in contrary to geometrically smooth and continuous seams of traditional seam carving [1]. However, it still required to scan the entire image to remove a single seam which is still time consuming. Huang et al. [24] presented a fast scheme by establishing an optimal adjacent relationship. Then they searched seams based on the adjacent relationship and removed seams by the minimum accumulated energy. The approach saved about $98\%-99\%$ time compared with the traditional seam carving [1], but Huang’s approach did not consider the problem of artifacts from seam removing. Wu [25] tried to alleviate the problem of artifacts and time consuming of seam carving simultaneously by improving Huang’s approach [24] with strip and neighboring probability constraints. However the strip partition did not consider content relationship between strips and the neighboring probability only improved the seam distribution in each strip. Wang et al. [26] proposed salient edge and region aware image retargeting approach to provide reliable salient edges and protect salient objects from distortion by seam carving. Although the discrete algorithms could preserve the details, the problem of time consuming and artifacts of seam removing are still remained.

Single operator-based resizing approaches are not optimally defined for all images and all target sizes. To generalize the scope, multiple operator-mixed hybrid retargeting algorithms become a promising way to guarantee the performance on various images and target sizes. The discrete algorithm usually preserves details but causes artifacts, however, the continuous algorithm generally causes small artifacts but produces the loss of details due to the use of interpolation. Therefore, these two kinds of algorithms are complementary in nature, which encourages a hybrid framework to combine the discrete and continuous algorithms [9]. Rubinstein et al. [27] presented an multi-operator (Multiop) image resizing algorithm to combine operators of bi-cubic scaling, cropping and seam carving in an optimal manner. Dong et al. [28] proposed IMED to combine seam carving and scaling coherently. They further combined DCD with image energy to construct another hybrid scheme [29]. SSPF in [30] presented an image retargeting approach employing seam searching and pixel fusion over scaling map. SSPF searched continuous seams by traditional seam carving method [8] and generated scaling map from the seams for pixel fusion. Fang et al. [31] proposed saliency detection in compressed domain and employed multi-operation of block-based seam carving and scaling to resize images. Moreover, they proposed multi-operator retargeting algorithm with retargeting operator sequence of seam carving, cropping, warping, and scaling iteratively [32]. The method arranged the order of operators with dense sift based structural similarity in iterations. Wu [33] proposed Deformation of semantic Edge (DSE) to jointly use seam carving with warping for sport image resizing. However, it is not stable and practical because the computation of DSE depends on the semantic edge detection. Therefore, an efficient hybrid approach with little distortion is still needed. These approaches generally perform better in image quality than the single operator based ones. However, due to the mixing of multiple operators all these approaches have high computational complexity and they are generally time consuming.

Wu et al. [34] proposed a fast hybrid framework in which fast seam carving with neighboring probability constraints (FSc_NeiP) was switched to scaling by the fast content-aware image distance. Seam carving in each strip did not consider content relationships between strips. It tended to result in different seam distribution in different strips containing contents of the same object (examples marked in yellow rectangles of Fig. 1(b)). It would cause artifacts to the object by removing seams. In Fig. 1(b), the cat ear, body and feet are segmented into different strips and the removed seams distribute differently between the strips. In the corresponding result of Fig. 1(d), the cat ears and cat feet are cut off a lot while the whole cat body is almost preserved, where the cat looks strange and distorts a lot from the original one. In addition, FSc searched the seams in the neighboring region and it can not avoid the seams passing through the important region which often distorted the key objects. Moreover, the neighboring probability constraints made seam distribution far from the strip boundary which produced apparent scene discontinuity and artifacts around the strip boundaries.

Inspired by [34], we propose a hybrid retargeting framework called fast non-continuous seam carving with seam context (FN2SC) as illustrated in Fig. 2. For an input image, the content aware strip partition is designed to segment the image into several strips over saliency estimation. The traditional equal spaced strip partition as Fig. 1(b) is updated by the bounding boxes of key objects as Fig. 1(c) to alleviate distortions to key objects as Fig. 1(e). The target size of each strip is estimated by their importance. For each strip, FN2SC conducts seam searching, seam context computing, seam removing and scaling switched by FCAID. In seam searching, all the non-continuous seams are searched by maximum correlation [24] in image scanning once. With the corresponding seam correlations, the neighboring probability (NeiP) and the touch bound probability (TBP) are formed as seam context to control the removed seams distributing scattered and resulting in less artifacts. In seam removing, the seams with the minimum energy are removed constrained by seam context. After a seam is removed, the energy of the removed seam is transferred to the seams in its neighborhood by NeiP and to the boundaries by TBP. If image distortion measured by FCAID exceeds the given threshold as FCAID > C_th, the seam removing of the current strip is switched to scaling the strip to the target size. The main contributions of this work are summarized as follows:

• FN2SC considers seam context of NeiP and TBP to constrain removed seams taking a scattered distribution, which relieves artifacts on key objects from concentrated seam removing.

• FN2SC employs content aware strip partition and estimate the target size of these strips by their average importance values. It helps make seams in the key object have relatively uniform distribution and alleviate distortions to the key objects in the retargeting result.

• FN2SC employs non-continuous seams to avoid the removed seams passing through the key objects and further relieve artifacts to the key objects.

• Both fast non-continuous seam searching and FCAID for switching in FN2SC aid effectively to deal with the time consuming problem of hybrid schemes.

Experimental results demonstrate that the proposed FN2SC is capable to resize images effectively with fewer artifacts and distortions to the key objects. Furthermore its efficiency is verified by the observations that FN2SC takes relatively small time cost to resize the images compared with its comparisons.

The remainder of this paper is organized as follows. Section 2 analyzes the reason that traditional seam carving causes artifacts. Section 3 introduces the details of content aware strip partition. Fast non-continuous seam carving is described in detail with the constraints of seam context information in Section 4. Section 5 provides a fast implementation to calculate the switching rule in the hybrid retargeting scheme. Section 7 provides experimental results and their corresponding analysis. We conclude this work in Section 8.