Abstract
Image harmonization aims to generate composite images that are visually consistent by adjusting the foreground to be compatible with the background. However, previous image harmonization methods overlook the fact that in a real image, the appearance (e.g., illumination, color temperature, saturation, hue, and texture) of different regions can vary significantly depending on content and position. For each foreground region, the background regions related to it should be taken as major references to adjust its appearance. To address this, a fine-grained appearance translation strategy is designed in this work. When adjusting the appearance of each foreground region, our method pays more attention to the background regions that are more relevant to it based on content similarity and position information. Furthermore, a multi-scale feature calibration strategy is introduced to adaptively calibrate the fine-grained features. Finally, an adaptive reconstruction strategy is proposed to further improve the harmonization result. Extensive experiments show our method significantly reduces parameters and achieves state-of-the-art performance compared with previous methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Xue, S., Agarwala, A., Dorsey, J., et al.: Understanding and improving the realism of image composites. ACM Trans. Graph. (TOG) 31(4), 1–10 (2012)
Cun, X., Pun, C.M.: Improving the harmony of the composite image by spatial-separated attention module. IEEE Trans. Image Process. 29, 4759–4771 (2020)
Zhang, L., Wen, T., Min, J., et al.: Learning object placement by inpainting for compositional data augmentation. In: ECCV, pp. 566–581 (2020)
Tsai, Y.H., Shen, X., Lin, Z., et al.: Deep image harmonization. In: CVPR, pp. 2799–2807 (2017)
Sofiiuk, K., Popenova, P., Konushin, A.: Foreground-aware semantic representations for image harmonization. In: WACV, pp. 1619–1628 (2021)
Cong, W., Zhang, J., Niu, L., et al.: DoveNet: deep image harmonization via domain verification. In: CVPR, pp. 8391–8400 (2020)
Cong, W., Niu, L., Zhang, J., et al.: BargainNet: background-guided domain translation for image harmonization. In: ICME, pp. 1–6 (2021)
Guo, Z., Zhang, H., Jiang, Y., et al.: Intrinsic image harmonization. In: CVPR, pp. 16362–16371 (2021)
Guo, Z., Gao, D., Zhang, H., et al.: Image harmonization with transformer. In: ICCV, pp. 14850–14859 (2021)
Guo, Z., Gou, Z., Zhang, B., et al.: Transformer for image harmonization and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 45, 12960–12977 (2022)
Ling, J., Xue, H., Song, L., et al.: Region-aware adaptive instance normalization for image harmonization. In: CVPR, pp. 9357–9366 (2021)
Zhu, Z., Zhang, Z., Lin, Z., et al.: Image harmonization by matching regional references. arXiv preprint arXiv:2204.04715 (2022)
Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015)
Reinhard, E., Adhikhmin, M., Gooch, B., et al.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)
Jia, J., Sun, J., Tang, C.K., et al.: Drag-and-drop pasting. ACM Trans. Graph. (TOG) 25(3), 631–637 (2006)
Tao, M.W., Johnson, M.K., Paris, S., et al.: Error-tolerant image compositing. Int. J. Comput. Vision 103, 178–189 (2013)
Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. (TOG) 22(3), 313–318 (2003)
Sunkavalli, K., Johnson, M.K., Matusik, W., et al.: Multi-scale image harmonization. ACM Trans. Graph. (TOG) 29(4), 1–10 (2010)
Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: NeurIPS, pp. 6000–6010 (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI, pp. 234–241 (2015)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV, pp. 1501–1510 (2017)
Ulyanov, D., Vedaldi, A., Lempitsky, Y.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
Liu, Z., Lin, Y., Cao, Y., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: ICCV, pp. 9992–10002 (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Acknowledgements.
This work was supported by the Natural Science Foundation of China (62276242), National Aviation Science Foundation (2022Z071078001), CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-016B, CAAIXSJLJJ-2022-001A), Anhui Province Key Research and Development Program (202104a05020007), USTC-IAT Application Sci. & Tech. Achievement Cultivation Program (JL06521001Y), Sci. & Tech. Innovation Special Zone (20-163-14-LZ-001-004-01).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ju, L., Pu, C., Gao, F., Yu, J. (2023). Adaptive Fine-Grained Region Matching for Image Harmonization. In: Lu, H., et al. Image and Graphics . ICIG 2023. Lecture Notes in Computer Science, vol 14357. Springer, Cham. https://doi.org/10.1007/978-3-031-46311-2_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-46311-2_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46310-5
Online ISBN: 978-3-031-46311-2
eBook Packages: Computer ScienceComputer Science (R0)