Skip to main content
SpringerLink
Log in

Image recoloring based on fast and flexible palette extraction

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Many operations, such as color transfer, recoloring, and image decomposition, are involved in the color manipulation of image editing. This paper introduces a simple, intuitive and interactive tool that allows non-experts to recolor an image by editing a color palette. First, we introduce a network that extracts a color palette from an image. Then, we introduce a network that decomposes the image into weighted layers, each corresponding to a palette color. In addition, a method is proposed to generate a variable-size palette and corresponding weighted layers. Applications such as image recoloring can be performed based on the final palette combined with the weighted layers. Our experiments show that it is much faster than the state-of-the-art image recoloring methods while keeping a better visual effect. Our algorithm can extract the palette from a 256×256 image in 0.077 s with the smallest palette loss 0.052 and then recolor the image in 0.3 s average.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Afifi M, Brubaker MA, Brown MS (2021) Histogan: Controlling colors of gan-generated and real images via color histograms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7941–7950

  2. Aharoni-Mack E, Shambik Y, Lischinski D (2017) Pigment-based recoloring of watercolor paintings. In: Proceedings of the Symposium on Non-Photorealistic Animation and Rendering - NPAR ’17. ACM Press. https://doi.org/10.1145/3092919.3092926

  3. Aksoy Y, Aydin TO, Smolić A., Pollefeys M (2017) Unmixing-based soft color segmentation for image manipulation. ACM Trans Graph 36(2):1–19. https://doi.org/10.1145/3002176

    Article  Google Scholar 

  4. Ahmed ST, Sreedhar Kumar S, Anusha B, Bhumika P, Gunashree M, Ishwarya B (2018) A generalized study on data mining and clustering algorithms. In: International Conference On Computational Vision and Bio-Inspired Computing, pp. 1121-1129. Springer. https://doi.org/10.1007/978-3-030-41862-5_114

  5. Bychkovsky V, Paris S, Chan E, Durand F (2011) Learning photographic global tonal adjustment with a database of input / output image pairs. In: The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition

  6. Chang H, Fried O, Liu Y, DiVerdi S, Finkelstein A (2015) Palette-based photo recoloring. ACM Trans Graph 34(4):1–11. https://doi.org/10.1145/2766978

    Article  Google Scholar 

  7. Chen J, Adams A, Wadhwa N, Hasinoff SW (2016) Bilateral guided upsampling. ACM Trans Graph 35(6):1–8. https://doi.org/10.1145/2980179.2982423

    Article  Google Scholar 

  8. Chen X, Zou D, Li J, Cao X, Zhao Q, Zhang H (2014) Sparse dictionary learning for edit propagation of high-resolution images. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. https://doi.org/10.1109/cvpr.2014.365

  9. Cho J, Yun S, Lee K, Choi JY (2017) Palettenet: Image recolorization with given color palette. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1058–1066. https://doi.org/10.1109/CVPRW.2017.143

  10. Ci Y, Ma X, Wang Z, Li H, Luo Z (2018) User-guided deep anime line art colorization with conditional adversarial networks. In: Proceedings of the 26th ACM International Conference on Multimedia. ACM. https://doi.org/10.1145/3240508.3240661

  11. Dubey SR, Chakraborty S, Roy SK, Mukherjee S, Singh SK, Chaudhuri BB (2020) diffGrad: An optimization method for convolutional neural networks. IEEE Trans Neural Netw Learn Syst 31 (11):4500–4511. https://doi.org/10.1109/tnnls.2019.2955777

    Article  MathSciNet  Google Scholar 

  12. Gupta RK, Chia A.Y.-S., Rajan D, Ng ES, Zhiyong H (2012) Image colorization using similar images. In: Proceedings of the 20th ACM International Conference on Multimedia - MM ’12. ACM Press. https://doi.org/10.1145/2393347.2393402

  13. Iizuka S, Simo-Serra E, Ishikawa H (2016) Let there be color!. ACM Trans Graph 35(4):1–11. https://doi.org/10.1145/2897824.2925974

    Article  Google Scholar 

  14. Kim S, Choi S (2021) Dynamic closest color warping to sort and compare palettes. ACM Trans Graph 40(4):1–15. https://doi.org/10.1145/3450626.3459776

    Article  Google Scholar 

  15. Levin A, Lischinski D, Weiss Y (2004) Colorization using optimization. ACM Trans Graph 23(3): 689–694. https://doi.org/10.1145/1015706.1015780

    Article  Google Scholar 

  16. Li Y, Ju T, Hu S.-M. (2010) Instant propagation of sparse edits on images and videos. Comput Graphics Forum 29(7):2049–2054

    Article  Google Scholar 

  17. Li Z, Zha Z, Cao Y (2019) Deep palette-based color decomposition for image recoloring with aesthetic suggestion. In: MultiMedia Modeling, pp. 127–138. Springer. https://doi.org/10.1007/978-3-030-37731-1_11

  18. Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models

  19. Matsui Y (2015) Challenge for manga processing. In: Proceedings of the 23rd ACM International Conference on Multimedia. ACM. https://doi.org/10.1145/2733373.2807997

  20. Morse BS, Thornton D, Xia Q, Uibel J (2007) Image-based color schemes. In: 2007 IEEE International Conference on Image Processing. IEEE. https://doi.org/10.1109/icip.2007.4379355

  21. O’Donovan P, Agarwala A, Hertzmann A (2011) Color compatibility from large datasets. ACM Trans Graph 30(4):1–12. https://doi.org/10.1145/2010324.1964958

    Article  Google Scholar 

  22. Phan HQ, Fu H, Chan AB (2018) Color orchestra: Ordering color palettes for interpolation and prediction. IEEE Trans Vis Comput Graph 24 (6):1942–1955. https://doi.org/10.1109/tvcg.2017.2697948

    Article  Google Scholar 

  23. Reinhard E, Adhikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput Graphics Appl 21(4):34–41. https://doi.org/10.1109/38.946629

    Article  Google Scholar 

  24. Shi Y, Chen S, Liu P, Long J, Cao N (2022) Colorcook: Augmenting color design for dashboarding with domain-associated palettes. Proc ACM Hum Comput Interact. 6(CSCW2). https://doi.org/10.1145/3555534

  25. Tan J, Echevarria J, Gingold Y (2018) Efficient palette-based decomposition and recoloring of images via RGBXY-space geometry. ACM Trans Graph 37(6):1–10. https://doi.org/10.1145/3272127.3275054

    Article  Google Scholar 

  26. Tan J, Lien J.-M., Gingold Y (2017) Decomposing images into layers via RGB-space geometry. ACM Trans Graph 36(1):1–14. https://doi.org/10.1145/2988229

    Article  Google Scholar 

  27. Wah C, Branson S, Welinder P, Perona P, Belongie S (2011) The caltech-ucsd birds-200-2011 dataset. california institute of technology

  28. Wang X, Song Y, Chen Y, Su C, Hu D (2021) ColorEmo: Color hunt with affective words. Wirel Commun Mob Comput 2021:1–9. https://doi.org/10.1155/2021/5589711

    Article  Google Scholar 

  29. Wang Y, Xia M, Qi L, Shao J, Qiao Y (2022) PalGAN: Image colorization with palette generative adversarial networks. In: Lecture Notes in Computer Science, pp. 271–288. Springer. https://doi.org/10.1007/978-3-031-19784-0_16

  30. X. D., Y. X., L. D., X. Y. (2012) Colorization using quaternion algebra with automatic scribble generation. In: Lecture Notes in Computer Science, pp. 103–114. Springer. https://doi.org/10.1007/978-3-642-27355-1_12

  31. Yang J, Chen Y, Westland S, Xiao K (2020) Predicting visual similarity between colour palettes. Color Res Appl 45 (3):401–408. https://doi.org/10.1002/col.22492

    Article  Google Scholar 

  32. Zhang Q, Nie Y, Zhu L, Xiao C, Zheng W.-S. (2022) A blind color separation model for faithful palette-based image recoloring. IEEE Trans Multimedia 24:1545–1557. https://doi.org/10.1109/TMM.2021.3067463

    Article  Google Scholar 

  33. Zhang Q, Xiao C, Sun H, Tang F (2017) Palette-based image recoloring using color decomposition optimization. IEEE Trans Image Process 26 (4):1952–1964. https://doi.org/10.1109/tip.2017.2671779

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We thank Yu Zhang for her help with the experiments and Zeeshan Tahir for his help to correct the language. This work was supported by Beijing Dailybread Co., Ltd.; the National Key Research and Development Program of China (2021YFF0600203); and the Major Scientific Research Project of the Zhejiang Laboratory (2022MG0AC04).

Funding

The authors declare they have no financial interests.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Zhejiang University, Hangzhou, China

    Simin Yan & Sanyuan Zhang

  2. College of Information Science and Technology, Hangzhou Normal University, Hangzhou, China

    Shuchang Xu

  3. Institute of Intelligent Perception Research, Zhejiang Lab, Hangzhou, China

    Wenzhen Yang

Authors
  1. Simin Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuchang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenzhen Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sanyuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shuchang Xu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, S., Xu, S., Yang, W. et al. Image recoloring based on fast and flexible palette extraction. Multimed Tools Appl 82, 47793–47810 (2023). https://doi.org/10.1007/s11042-023-15114-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15114-5

Keywords

Search

Navigation