Selective color transferring via ellipsoid color mixture map

doi:10.1016/j.jvcir.2011.09.006

Journal of Visual Communication and Image Representation

Volume 23, Issue 1, January 2012, Pages 173-181

https://doi.org/10.1016/j.jvcir.2011.09.006 Get rights and content

Abstract

Color transfer among images is a natural phenomenon and important research topic in interactive graphics and augmented environments. Most of the conventional methods need complex user interactions and difficult image segmentation. Motivated for the intuitive use of color editing tools, this paper presents a simple but effective ellipsoid color mixture map to realize selective color transfers. Our approach proposes ellipsoid hulls to represent color statistics of the images. Based on the window-click input, the system computes the ellipsoid hulls of the source and target images respectively. The color mixture map is generated to determine the blending weight of pixels in the output image, according to the color and distance information instead of using image segmentation. By mixing the images using the color mixture map, the final results we produce have the source color selected realistically spreading on the structures related to the target window. Our selective color transferring approach is efficient and simple to use, and widely applicable for images and also video sequences without the need of addition interaction. Our experimental results have showed the high-quality visual effects and efficiency of image/video color transfers.

Highlights

► The ellipsoid hulls are proposed to express the color statistics of images. ► Color transfer among images is converted to simple transform of ellipsoids. ► It can efficiently perform selective color transferring of images and videos.

Introduction

Color transfer among images/video is the processing to change the color of related structures in output image according to the reference color in source image. It is a common practice and also important digital-media editing tool to alter the coloring appearance, which has a variety of applications in artistic design, image/video enhancement, special effects modeling, interactive multi-media production, and augmented reality.

Previous work focused on global color transfer which affected the coloring of the whole image. However, in many cases, the user would selectively edit the color of related structures while maintaining the appearance of the other parts unchanged in the output image. Most of the methods are based on the pioneer work of Reinhard et al. [1], which proposed for global color transfer. To realize local color transfer, color segmentation and clustering methods are used in conventional methods, which are often challenging tasks in image processing. Users may need to input strokes to select the area of interest. It is not practical for non-professionals to use such systems to interactively edit the images/video as they simply intended.

In this paper, we present a novel color transferring using ellipsoid color mixture maps to selectively re-render the colors. Once the user simply selects the color by a window-click in the source image then the target window in the output image, the system we develop automatically transfers the source color selected to the related structures of the output image. Our approach can be efficiently applied for images and also video sequences without additional user interaction. The main contributions of our work are the following:

•
The ellipsoid hulls are proposed to express the color statistics of images, and color transfer among images is converted to the transform of ellipsoids which is simple in geometry analysis.
•
Our approach can efficiently perform selective color transferring of images and video sequences, locally or globally. It is the first attempt to perform selective color transfer of videos efficiently with the related structures.
•
By the simple window-click interaction, our system automatically searches the object of interest for high-quality color transfers, mostly suitable for interactive rich-media applications.

The rest of the paper is organized as follows: Section 2 outlines the brief review of related work in image color transfer. Section 3 presents the processing framework of our approach, including ellipsoid hulls, the intermediate image and color mixture map used for selective color transferring of images/video. Section 4 shows the experimental results using our SCT (selective color transferring) approach in image/video color transfers, and testing performance of control parameters and rendering rates. Finally, summary and further work go to Section 5.

Section snippets

Related work

Researches related to color alteration recently in graphics and image processing include color transfer [1], [2], [3], colorization [4], [5], [6], [7], tone adjustment [8], [9], [10], [11]. Here, we mainly review the previous work for color transfer mainly global or local methods related to our SCT approach.

As the pioneer work, Reinhard et al. [1] proposed a simple but very successful method that transferred color characteristics from the source to the target images. It transformed pixels’

Selective color transferring

When performing local color transfer, users need to select a region of interest initially. This can be produced based on the image segmentation, which however has high computation complexity and can not well reflect the user’s intention. Interactively using many strokes to select the accurate contour of the region is usually hard, and not a good choice for non-professional users. In our approach, users only input a window click to simply indicate the region of interest in the images

SCT results

We have developed our SCT approach, and performed experimental tests of color transferring for natural images/videos run on an Intel P8600 CPU with 4.0 GB RAM. Fig. 4 shows the SCT testing example of a flower scene, where once the user inputs the window clicks in the source and target images, our system performs the selective color transferring automatically with vivid visual effects. Fig. 4(c) shows the color mixture map calculated according to the window clicks, and (d) the final result that

Summary

Color transfer is the useful and also important digital-media editing tool to alter the coloring appearance in natural images/video. In this paper, we present the novel and simple color transferring using ellipsoid color mixture map, for the users to selectively re-render the colors. Upon the window clicks, our system automatically computes the related structures and then transfers the selected color to the target image with the mimic visual effects, locally or globally. To realize the

Acknowledgements

We thank authors of [2], [3], [21] for their kind help to make comparisons. We also thank the anonymous reviewers for their valuable comments. The research was supported by National Science Foundation of China (No. 61170118 and 60803047), the Specialized Research Fund for the Doctoral Program of Higher Education of China (No. 200800561045).

References (26)

A. Abadpour et al.
An efficient PCA-based color transfer method
Journal of Visual Communication and Image Representation
(2007)
Y. Xiang et al.
Selective color transfer with multi-source images
Pattern Recognition Letter
(2009)
E. Reinhard et al.
Color transfer between images
IEEE Computer Graphics and Applications.
(2001)
Q. Luan, F. Wen, Y. Xu, Color transfer brush, in: Proceedings of 15th Pacific Conference on Computer Graphics and...
C. Wen et al.
Example-based multiple local color transfer by strokes
Computer Graphics Forum
(2008)
A. Levin et al.
Colorization using optimization
ACM Transactions on Graphics
(2004)
R. Irony, D. Cohen-Or, D. Lischinski, Colorization by example, in: Proceedings of Eurographics Symposium on Rendering,...
Y. Qu et al.
Manga colorization
ACM Transactions on Graphics
(2006)
L. Yatziv et al.
Fast image and video colorization using chrominance blending
IEEE Transaction on Image Processing
(2006)
P. Perez et al.
Poisson image editing
ACM Transactions on Graphics
(2003)

S. Bae et al.

Two-scale tone management for photographic look

ACM Transaction on Graphics

(2006)

D. Lischinski et al.

Interactive local adjustment of tonal values

ACM Transaction on Graphics

(2006)

X. An et al.

AppProp: All-Pairs Appearance-Space Edit Propagation

ACM Transactions on Graphics

(2008)

Cited by (20)

Progressive complex illumination image appearance transfer based on CNN
2019, Journal of Visual Communication and Image Representation
Citation Excerpt :
Iizuka et al. [19] proposed a method to combine the global prior knowledge and local characteristics in the process of image colorization, which can deal with images with arbitrary resolution and obtain better colorization effect without any user interaction. Liu et al. [21] presented an effective ellipsoid color mixture map method to realize selective color transfer. Limmer et al. [20] proposed an infrared image colorization method based on DNN that achieved multi-scale convolution neural network learning by adopting pyramid images as input.
Image appearance transfer is the process of transferring the appearance features from the user-supplied reference image to the target image. Since traditional appearance transfer methods are based on low-level feature, it is difficult to obtain natural effect in the case of complex illumination between the target image and the reference image. This is mainly because that the appearance mapping relation with complex illumination is a highly nonlinear problem. Although traditional methods are good at dealing with linear transformations, they are less suitable for solving the highly nonlinear problems caused by such complex illumination. Moreover, convolutional neural network (CNN) can extract the hierarchical abstraction features at different levels of the network layer, so it can be conveniently used to realize progressive transfer. In this paper, we propose a novel method for progressive appearance transfer for images with complex illumination. Firstly, we convert the input images from the RGB color space to the HSV color space. The illumination transfer is carried out only in the illumination channel of the image, and for the other two channels, the color distribution of the reference image is transferred to the target image. Secondly, CNN is specially designed to extract the hierarchical feature maps. To achieve progressive transfer, the histogram reshaping method is carried out by using the hierarchical feature maps extracted from the CNN. The appearance transfer results are obtained after the illumination transfer and color transfer. To optimize the transfer results, we adopt the joint bilateral filter to smooth the noises. The experimental results show that our method can effectively solve the problem of progressive appearance transfer for images with complex illumination.
Local texture-based color transfer and colorization
2017, Computers and Graphics (Pergamon)
This paper targets two related color manipulation problems: Color transfer for modifying an image׳s colors and colorization for adding colors to a grayscale image. Automatic methods for these two applications propose to modify the input image using a reference that contains the desired colors. Previous approaches usually do not target both applications and suffer from two main limitations: possible misleading associations between input and reference regions and poor spatial coherence around image structures. In this paper, we propose a unified framework that uses the textural content of the images to guide the color transfer and colorization. Our method introduces an edge-aware texture descriptor based on region covariance, allowing for local color transformations. We show that our approach is able to produce results comparable or better than state-of-the-art methods in both applications.
Adaptive Multilevel Coloring and Significant Texture Selecting for Automatic Deep Learning Image Transfer
2022, Electronics (Switzerland)
Image recoloring of printed fabric based on the salient map and local color transfer
2022, Textile Research Journal
Two Decades of Colorization and Decolorization for Images and Videos
2022, arXiv
An Overview of Color Transfer and Style Transfer for Images and Videos
2022, arXiv

View all citing articles on Scopus

View full text

Selective color transferring via ellipsoid color mixture map

Abstract

Highlights

Introduction

Section snippets

Related work

Selective color transferring

SCT results

Summary

Acknowledgements

Journal of Visual Communication and Image Representation

Pattern Recognition Letter

Color transfer between images

IEEE Computer Graphics and Applications.

Example-based multiple local color transfer by strokes

Computer Graphics Forum

Colorization using optimization

ACM Transactions on Graphics

Manga colorization

ACM Transactions on Graphics

Fast image and video colorization using chrominance blending

IEEE Transaction on Image Processing

Poisson image editing

ACM Transactions on Graphics

Two-scale tone management for photographic look

ACM Transaction on Graphics

Interactive local adjustment of tonal values

ACM Transaction on Graphics

AppProp: All-Pairs Appearance-Space Edit Propagation

ACM Transactions on Graphics