research-article

Structure Guided Photorealistic Style Transfer

Authors:

Bingbing NiAuthors Info & Claims

MM '18: Proceedings of the 26th ACM international conference on Multimedia

Pages 365 - 373

https://doi.org/10.1145/3240508.3240637

Published: 15 October 2018 Publication History

Abstract

Recent style transfer methods based on deep networks strive to generate more content matching stylized images by adding semantic guidance in the iterative process. However, these approaches can just guarantee the transfer of integral color and texture distribution between semantically equivalent regions, but local variation within these regions cannot be accurately captured. Therefore, the resulting image lacks local plausibility. To this end, we develop a non-parametric patch based style transfer framework to synthesize more content coherent images. By designing a novel patch matching algorithm which simultaneously takes high-level category information and geometric structure information (e.g., human pose and building structure) into account, our proposed method is capable of transferring more detailed distribution and producing more photorealistic stylized images. We show that our approach achieves remarkable style transfer results on contents with geometric structure, including human body, vehicles, buildings, etc.

Supplementary Material

ZIP File (fp0623.zip)

We provide all the images used in the user study of this paper in this supplemental material, sorted by the corresponding method. During the actual user study procedure, however, the images produced in each scene were shuffled randomly and the corresponding method is not shown to the user.

Download
8.61 MB

References

[1]

Soonmin Bae, Sylvain Paris, and Frédo Durand. 2006. Two-scale tone management for photographic look. In ACM Transactions on Graphics (TOG), Vol. 25. ACM, 637--645.

Digital Library

[2]

PETER J BURT and EDWARD H ADELSON. 1983. The Laplacian Pyramid as a Compact Image Code. IEEE TRANSACTIONS ON COMMUNICATIONS, Vol. 3, 4 (1983).

[3]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In CVPR .

[4]

Alex J Champandard. 2016. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768 (2016).

[5]

Alexei A Efros and William T Freeman. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 341--346.

Digital Library

[6]

Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier. 2016. Split and match: Example-based adaptive patch sampling for unsupervised style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 553--561.

[7]

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2414--2423.

[8]

Joon-Young Lee, Kalyan Sunkavalli, Zhe Lin, Xiaohui Shen, and In So Kweon. 2016. Automatic content-aware color and tone stylization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2470--2478.

[9]

Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 30, 2 (2008), 228--242.

Digital Library

[10]

Chuan Li and Michael Wand. 2016. Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2479--2486.

[11]

Xiaodan Liang, Chunyan Xu, Xiaohui Shen, Jianchao Yang, Si Liu, Jinhui Tang, Liang Lin, and Shuicheng Yan. 2015. Human parsing with contextualized convolutional neural network. In Proceedings of the IEEE International Conference on Computer Vision. 1386--1394.

Digital Library

[12]

Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual attribute transfer through deep image analogy. ACM Transactions on Graphics (TOG), Vol. 36, 4 (2017), 120.

Digital Library

[13]

Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep Photo Style Transfer. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 6997--7005.

[14]

Francois Pitie, Anil C Kokaram, and Rozenn Dahyot. 2005. N-dimensional probability density function transfer and its application to color transfer. In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, Vol. 2. IEEE, 1434--1439.

Digital Library

[15]

Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer graphics and applications, Vol. 21, 5 (2001), 34--41.

Digital Library

[16]

Ahmed Selim, Mohamed Elgharib, and Linda Doyle. 2016. Painting style transfer for head portraits using convolutional neural networks. ACM Transactions on Graphics (ToG), Vol. 35, 4 (2016), 129.

Digital Library

[17]

YiChang Shih, Sylvain Paris, Connelly Barnes, William T Freeman, and Frédo Durand. 2014. Style transfer for headshot portraits. (2014).

[18]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

Cited By

An JLi THuang HMa JLuo J(2023)Is Bigger Always Better? An Empirical Study on Efficient Architectures for Style Transfer and Beyond2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00407(4073-4083)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00407
Yang YLiang HYang YXu W(2021)Image Arbitrary Style Transfer via Self-Attention Mechanism Based on Feature Fusion2021 2nd International Conference on Artificial Intelligence and Education (ICAIE)10.1109/ICAIE53562.2021.00019(58-63)Online publication date: Jun-2021
https://doi.org/10.1109/ICAIE53562.2021.00019
Deng YTang FDong WSun WHuang FXu CWen Chen CCucchiara RHua XQi GRicci EZhang ZZimmermann R(2020)Arbitrary Style Transfer via Multi-Adaptation NetworkProceedings of the 28th ACM International Conference on Multimedia10.1145/3394171.3414015(2719-2727)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3394171.3414015
Show More Cited By

Index Terms

Structure Guided Photorealistic Style Transfer
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Image manipulation

Recommendations

Aesthetic-Aware Image Style Transfer
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Style transfer aims to synthesize an image which inherits the content of one image while preserving a similar style of the other one. The "style'' of an image usually refers to its unique feeling conveyed from visual features, which is highly related to ...
Structure-Aware Photorealistic Style Transfer Using Ghost Bottlenecks
Pattern Recognition and Artificial Intelligence
Abstract
Photorealistic style transfer synthesizes a new image from a pair of content and style images. The transfer should express the visual patterns in the former while preserving the content details following the latter. However, most existing methods ...
Semantic Correspondence Guided Deep Photo Style Transfer
Advances in Multimedia Information Processing – PCM 2018
Abstract
The objective of this paper is to develop an effective photographic transfer method while preserving the semantic correspondence between the style and content images. A semantic correspondence guided deep photo style transfer algorithm is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '18: Proceedings of the 26th ACM international conference on Multimedia

October 2018

2167 pages

ISBN:9781450356657

DOI:10.1145/3240508

General Chairs:
Susanne Boll
University of Oldenburg, Germany
,
Kyoung Mu Lee
Seoul National University, Korea
,
Jiebo Luo
University of Rochester, USA
,
Wenwu Zhu
Tsinghua University, China
,
Program Chairs:
Hyeran Byun
Yonsei University, Korea
,
Chang Wen Chen
State Univ. Of New York at Buffalo, USA
,
Rainer Lienhart
University of Augsburg, Germany
,
Tao Mei
JD AI, China

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China
National Natural Science Foundation of China
SJTU-UCLA Joint Center for Machine Perception and Inference
Science and Technology Commission of Shanghai Municipality
China's Thousand Youth Talents Plan

Conference

MM '18

Sponsor:

SIGMM

MM '18: ACM Multimedia Conference

October 22 - 26, 2018

Seoul, Republic of Korea

Acceptance Rates

MM '18 Paper Acceptance Rate 209 of 757 submissions, 28%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
586
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

An JLi THuang HMa JLuo J(2023)Is Bigger Always Better? An Empirical Study on Efficient Architectures for Style Transfer and Beyond2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00407(4073-4083)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00407
Yang YLiang HYang YXu W(2021)Image Arbitrary Style Transfer via Self-Attention Mechanism Based on Feature Fusion2021 2nd International Conference on Artificial Intelligence and Education (ICAIE)10.1109/ICAIE53562.2021.00019(58-63)Online publication date: Jun-2021
https://doi.org/10.1109/ICAIE53562.2021.00019
Deng YTang FDong WSun WHuang FXu CWen Chen CCucchiara RHua XQi GRicci EZhang ZZimmermann R(2020)Arbitrary Style Transfer via Multi-Adaptation NetworkProceedings of the 28th ACM International Conference on Multimedia10.1145/3394171.3414015(2719-2727)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3394171.3414015
Ratajczak RCrispim-Junior CFervers BFaure ETougne LGouet-Brunet VKhokhlova MChen LMünster S(2019)Pseudo-Cyclic Network for Unsupervised Colorization with Handcrafted Translation and Output Spatial PyramidsProceedings of the 1st Workshop on Structuring and Understanding of Multimedia heritAge Contents10.1145/3347317.3357243(5-13)Online publication date: 15-Oct-2019
https://dl.acm.org/doi/10.1145/3347317.3357243

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten