Abstract
This paper proposes a structure-aware nonlocal energy optimization framework for interactive image colorization with sparse scribbles. Our colorization technique propagates colors to both local intensity-continuous regions and remote texture-similar regions without explicit image segmentation. We implement the nonlocal principle by computing k nearest neighbors in the high-dimensional feature space. The feature space contains not only image coordinates and intensities but also statistical texture features obtained with the direction-aligned Gabor wavelet filter. Structure maps are utilized to scale texture features to avoid artifacts along high-contrast boundaries. We show various experimental results and comparisons on image colorization, selective recoloring and decoloring, and progressive color editing to demonstrate the effectiveness of the proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Reinhard E, Ashikhmin M, Gooch B, Shirley P. Color transfer between images. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41.
Chang Y, Saito S, Nakajima M. Example-based color transformation of image and video using basic color categories. IEEE Trans. Image Processing, 2007, 16(2): 329-336.
Xiao X, Ma L. Gradient-preserving color transfer. Computer Graphics Forum, 2009, 28(7): 1879-1886.
Levin A, Lischinski D, Weiss Y. Colorization using optimization. ACM Trans. Graphics, 2004, 23(3): 689-694.
Sheng B, Sun H, Chen S, Liu X, Wu E. Colorization using the rotation-invariant feature space. IEEE Computer Graphics Applications, 2011, 31(2): 24-35.
Yatziv L, Sapiro G. Fast image and video colorization using chrominance blending. IEEE Transactions on Image Processing, 2006, 15(5): 1120-1129.
Qu Y, Wong T T, Heng P A. Manga colorization. ACM Transactions on Graphics, 2006, 25(3): 1214-1220.
Luan Q, Wen F, Cohen-Or D, Liang L, Xu Y Q, Shum H Y. Natural image colorization. In Proc. the 18th Eurographics Workshop on Rendering, July 2007, pp.309-320.
Kyprianidis J E, Kang H. Image and video abstraction by coherence-enhancing filtering. Computer Graphics Forum, 2011, 30(2): 593-602.
Manjunath B S, Ma W Y. Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 18(8): 837-842.
Xu L, Yan Q, Xia Y, Jia J. Structure extraction from texture via relative total variation. ACM Transactions on Graphics, 2012, 31(6): 139:1-139:10.
Welsh T, Ashikhmin M, Mueller K. Transferring color to greyscale images. ACM Trans. Graphics, 2002, 21(3): 277-280.
Ironi R, Cohen-Or D, Lischinski D. Colorization by example. In Proc. the Eurographics Symposiums on Rendering Techniques, June 29-July 1, 2005, pp.201-210.
Cohen-Or D, Sorkine O, Gal R, Leyvand T, Xu Y Q. Color harmonization. ACM Trans. Graphics, 2006, 25(3): 624-630.
Liu X, Wan L, Qu Y et al. Intrinsic colorization. ACM Trans. Graphics, 2008, 27(5): 152:1-152:9.
Chia A Y S, Zhuo S, Gupta R K et al. Semantic colorization with internet images. ACM Trans. Graphics, 2011, 30(6): 156:1-156:8.
An X, Pellacini F. AppProp: All-pairs appearance-space edit propagation. ACM Trans. Graphics, 2008, 27(3): 40:1-40:9.
Fattal R. Edge-avoiding wavelets and their applications. ACM Trans. Graphics, 2009, 28(3): Article No. 22.
Xu K, Li Y, Ju T, Hu S M, Liu T Q. Efficient affinity-based edit propagation using k-d tree. ACM Transactions on Graphics, 2009, 28(5): Article No. 118.
Bhat P, Zitnick C L, Cohen M et al. GradientShop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graphics, 2010, 29(2): 10:1-10:14.
Musialski P, Cui M, Ye J et al. A framework for interactive image color editing. The Visual Computer, 2013, 29(11): 1173-1186.
Huang H, Li X, Zhao H et al. Manifold-preserving image colorization with nonlocal estimation. Multimedia Tools and Applications, 2014. DOI: 10.1007/s11042-014-1991-5.
Jeschke S, Cline D, Wonka P. A GPU Laplacian solver for diffusion curves and Poisson image editing. ACM Transactions on Graphics, 2009, 28(5): 116:1-116:8.
Canny J. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986, PAMI-8(6): 679-698.
Chen Q, Li D, Tang C K. KNN matting. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, June 2012, pp.869-876.
Buatois L, Caumon G, Levy B. Concurrent number cruncher: A GPU implementation of a general sparse linear solver. International Journal of Parallel, Emergent and Distributed Systems, 2009, 24(3): 205-223.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by the National Natural Science Foundation of China under Grant Nos. 61100146 and 61472351, and the Zhejiang Provincial Natural Science Foundation of China under Grant Nos. LY15F020019 and LQ14F020006. Pan was supported by the National Key Technology Research and Development Program of the Ministry of Science and Technology of China under Grant No. 2013BAH24F01.
Rights and permissions
About this article
Cite this article
Zhao, HL., Nie, GZ., Li, XJ. et al. Structure-Aware Nonlocal Optimization Framework for Image Colorization. J. Comput. Sci. Technol. 30, 478–488 (2015). https://doi.org/10.1007/s11390-015-1538-x
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11390-015-1538-x