Skip to main content
SpringerLink
Log in

Deep color transfer using histogram analogy

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We propose a novel approach to transferring the color of a reference image to a given source image. Although there can be diverse pairs of source and reference images in terms of content and composition similarity, previous methods are not capable of covering the whole diversity. To resolve this limitation, we propose a deep neural network that leverages color histogram analogy for color transfer. A histogram contains essential color information of an image, and our network utilizes the analogy between the source and reference histograms to modulate the color of the source image with abstract color features of the reference image. In our approach, histogram analogy is exploited basically among the whole images, but it can also be applied to semantically corresponding regions in the case that the source and reference images have similar contents with different compositions. Experimental results show that our approach effectively transfers the reference colors to the source images in a variety of settings. We also demonstrate a few applications of our approach, such as palette-based recolorization, color enhancement, and color editing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. Readers may also refer to the code that will be released.

References

  1. Arbelot, B., Vergne, R., Hurtut, T., Thollot, J.: Local texturebased color transfer and colorization. In: Computers & Graphics 62 (2017)

  2. Bychkovsky, V., Paris, S., Chan, E., Durand, F.: Learning photographic global tonal adjustment with a database of input/output image pairs. In: The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition (2011)

  3. Chang, H., Fried, O., Liu, Y., DiVerdi, S., Finkelstein, A.: Palette-based photo recoloring. In: ACM Transactions on Graphics (2015)

  4. Cho, J., Yun, S., Lee, K., Choi, J.: Palettenet: Image recolorization with given color palette. In: CVPR Workshops (2017)

  5. Freedman, D., Kisilev, P.: Object-to-object color transfer: optimal flows and smsp transformations. In: Proceedings of the CVPR (2010)

  6. Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. In: CoRR (2015)

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the CVPR (2016)

  8. He, M., Liao, J., Chen, D., Yuan, L., Sander, P.V.: Progressive color transfer with dense semantic correspondences. ACM Trans. Graph. 38(2), 13:1–13:18 (2019)

    Article  Google Scholar 

  9. Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of the SIGGRAPH (2001)

  10. Hristova, H., Meur, O.L., Cozot, R., , Bouatouch, K.: Style-aware robust color transfer. In: Proceedings of the workshop on Computational Aesthetics (2015)

  11. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the ICCV (2017)

  12. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of the ICLR (2014)

  13. Kingma, D.P., Ba, J.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) (2015)

  14. Lee, H.Y., Tseng, H.Y., Huang, J.B., Singh, M.K., Yang, M.H.: Diverse image-to-image translation via disentangled representations. In: Proceedings of the ECCV (2018)

  15. Lee, J., Sunkavalli, K., Lin, Z., Shen, X., Kweon, I.: Automatic content-aware color and tone stylization. In: Proceedings of the CVPR (2016)

  16. Li, X., Liu, S., Kautz, J., Yang, M.H.: Learning linear transformations for fast arbitrary style transfer. In: Proceedings of the CVPR (2019)

  17. Li, Y.: A closed-form solution to photorealistic image stylization. In: Proceedings of the ECCV (2018)

  18. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.: Universal style transfer via feature transforms (2017)

  19. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Universal style transfer via feature transforms. In: Proceedings of the NIPS (2017)

  20. Li, C., Vishwanathan, S.V.N., Xinhua, Z.: Consistent image analogies using semi-supervised learning. In: Proceedigs of the CVPR (2008)

  21. Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy. In: Proceedings of the SIGGRAPH (2017)

  22. Lin, S., Ritchie, D., Fisher, M., Hanrahan, P.: Probabilistic color-by-numbers: Suggesting pattern colorizations using factor graphs. In: ACM Transactions on Graphics (2013)

  23. Liu, S., Mello, S.D., Gu, J., Zhong, G., Yang, M., Kautz, J.: Learning affinity via spatial propagation networks. In: Proceedings of the NIPS (2017)

  24. Liu, Z., Li, X., Luo, P., Loy, C.C., Tang, X.: Deep learning Markov random field for semantic segmentation. In: TPAMI (2017)

  25. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the CVPR (2015)

  26. Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: Proceedings of the CVPR (2017)

  27. Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. In: https://distill.pub/2016/deconv-checkerboard/ (2016)

  28. O’Donovan, P., Agarwala, A., Hertzmann, A.: Color compatibility from large datasets. In: Proceedings of the SIGGRAPH (2011)

  29. Pitie, F., Kokaram, A., Dahyot, R.: N-dimensional probability density function transfer and its application to color transfer. In: Proceedings of the ICCV (2005)

  30. Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. In: IEEE Computer Graphics and Applications 21(5) (2001)

  31. Tai, Y.W., Jia, J., , Tang, C.K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: Proceedings of the CVPR (2005)

  32. Tsai, Y.H., Shen, X., Lin, Z., Sunkavalli, K., Yang, M.H.: Sky is not the limit: semantic-aware sky replacement. In: ACM Transactions on Graphics (2016)

  33. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)

  34. Wang, Z., Li, H., Ouyang, W., Wang, X.: Learnable histogram: statistical context features for deep neural networks (2016)

  35. Wonwoong Cho Sungha Choi, D.K.P.I.S.J.C.: Image-to-image translation via group-wise deep whitening-and-coloring transformation. In: Proceedings of the CVPR (2019)

  36. Yoo, J., Uh, Y., Chun, S., Kang, B., Ha, J.: Photorealistic style transfer via wavelet transforms. In: Proceedings of the ICCV (2019)

Download references

Acknowledgements

This work was supported by the Ministry of Science and ICT, Korea, through IITP Grants (SW Star Lab, IITP-2015-0-00174; Artificial Intelligence Graduate School Program (POSTECH), IITP-2019- 0-01906) and NRF Grant (NRF-2017M3C4A7066317).

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, POSTECH, Pohang, South Korea

    Junyong Lee, Hyeongseok Son, Jonghyeop Lee, Sunghyun Cho & Seungyong Lee

  2. NCSOFT, Seongnam, South Korea

    Gunhee Lee

Authors
  1. Junyong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyeongseok Son
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gunhee Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jonghyeop Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sunghyun Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Seungyong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seungyong Lee.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Network architecture details

1.1 Color transfer network (CTN)

Our proposed CTN adopts modified structures of U-Net [13] and ResNet [7]. CTN consists of four sub-modules: encoder, decoder, residual blocks, and auxiliary modules.Footnote 1

The encoder consists of five convolutional blocks, where each block consists of a varying number of convolutional layers followed by instance normalization [33] and leaky ReLU layers. The encoder and decoder are connected through skip connections, and the features produced by the encoder are passed to the decoder. The decoder also consists of five convolution blocks. To avoid checkerboard artifact [27], each convolutional block uses upsampling convolution, which performs convolution after bilinear upsampling, instead of using deconvolution. Then, the final feature map of the decoder goes into the residual blocks, which produce the final recolored image. In addition, for training stability, we attach auxiliary modules at the end of each convolutional block in the decoder to produce temporary result images of smaller scales.

1.2 Histogram encoding network (HEN)

HEN consists of eight convolution layers, each of which is followed by leaky ReLU activation. Given input and reference images, HEN generates an encoded histogram feature for each of input and reference images.

Appendix B: User study details

We conducted the user study using Amazon Mechanical Turk (AMT) with 20 participants whose hit approval rates are over 98%, and 30 source/reference image pairs from [15, 26]. The image pairs cover all three cases: strong relevance, weak relevance, and irrelevance, each of which has 10 pairs.

We compared our method with four state-of-the-art color transfer methods: Pitie et al. [29], Luan et al. [26], He et al. [8], and Yoo et al. [36], in a pair-wise manner. Specifically, we showed each participant all possible pairs of the results of different color transfer methods side-by-side as well as their original and reference images. As a result, each participant was shown 300 pairs of result images generated by five methods. Then, for each result image pair, we asked the participants three questions: (1) Which image better preserves the content of the original image? (2) which image better reflects the color of the reference? and (3) which image is better in terms of overall quality?

Table 4 shows the user study result in the form of preference matrices. Each element in the matrices shows the number of votes for one method (specified by its row) over another method (specified by its column). For instance, the element at the fifth row and second column of strong relevant case in Table 4 means that our results are preferred over the results of Luan et al. [26] by 68% (136 among 200 votes) in the aspect of content preservation in the case of strong relevance. As the tables illustrate, our method outperforms the others in terms of color reflection, content preservation, and overall quality. Figures 12 and 13 show some samples used in the survey.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, J., Son, H., Lee, G. et al. Deep color transfer using histogram analogy. Vis Comput 36, 2129–2143 (2020). https://doi.org/10.1007/s00371-020-01921-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01921-6

Keywords

Search

Navigation