Abstract
Photorealistic style transfer synthesizes a new image from a pair of content and style images. The transfer should express the visual patterns in the former while preserving the content details following the latter. However, most existing methods may generate an image that suffers disrupted content details and unexpected visual cues from the style image; hence, they do not satisfy the photorealism. We tackle this issue with a new style transfer architecture that effectively unifies a content encoder, an Xception style encoder, a Ghost Bottlenecks subnet, and a decoder. In our framework, the style features extracted from the Xception module balance well with the content features obtained from an encoder; the Ghost Bottlenecks subnet then integrates these features and feeds them into a decoder to produce the resulting style transferred image. Experimental results demonstrate that our model surmounts the structure distortion problem to satisfy photorealistic style transfer and hence obtains impressive visual effects.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., Xu, C.: GhostNet: more features from cheap operations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Heeger, D.J., Bergen, J.R.: Pyramid-based texture analysis/synthesis. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 229–238 (1995)
Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 341–346 (2001)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Chen, T.Q., Schmidt, M.: Fast Patch-Based Style Transfer of Arbitrary Style. arXiv preprint arXiv:1612.04337 (2016)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (ECCV) (2016)
Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.-H.: Universal style transfer via feature transforms. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS) (2017)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: IEEE International Conference on Computer Vision (ICCV) (2017)
Li, S., Xu, X., Nie, L., Chua, T.-S.: Laplacian-steered neural style transfer. In: Proceedings of the 25th ACM international conference on Multimedia (2017)
Cheng, M.-M., Liu, X.-C., Wang, J., Lu, S.-P., Lai, Y.-K., Rosin, P.L.: Structure-preserving neural style transfer. IEEE Trans. Image Process. 29, 909–920 (2020)
Ranzato, M., Huang, F.-J., Boureau, Y.-L., LeCun, Y.: Unsupervised learning of invariant feature hierarchies with applications to object recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2007)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: International Conference on Learning Representations (2014)
Goodfellow, I.J., et al.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), vol. 2, pp. 2672–2680 (2014)
Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. In: Artificial Intelligence Review (2020)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv preprint arXiv:1607.08022 (2016)
Li, X., Liu, S., Kautz, J., Yang, M.-H.: Learning linear transformations for fast image and video style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Huang, X., Liu, M.-Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. In: European Conference on Computer Vision (ECCV) (2018)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML), vol. 37, pp. 448–456 (2015)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: European Conference on Computer Vision (ECCV) (2014)
Nichol, K.: Painter by numbers, wikiart. https://www.kaggle.com/c/painter-by-numbers
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
Yao, Y., Ren, J., Xie, X., Liu, W., Liu, Y.-J., Wang, J.: Attention-aware multi-stroke style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Wang, H., Li, Y., Wang, Y., Hu, H., Yang, M.: Collaborative distillation for ultra-resolution universal style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Singh, A., Hingane, S., Gong, X., Wang, Z.: SAFIN: arbitrary style transfer with self-attentive factorized instance normalization. In: IEEE International Conference on Multimedia and Expo (ICME) (2021)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. In: IEEE Transactions on Image Processing, vol. 13, pp. 600–612 (2004)
Canny, J.: A computational approach to edge detection. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 679–698 (1986)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Bui, NT., Nguyen, NT., Cao, XN. (2022). Structure-Aware Photorealistic Style Transfer Using Ghost Bottlenecks. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13363. Springer, Cham. https://doi.org/10.1007/978-3-031-09037-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-09037-0_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09036-3
Online ISBN: 978-3-031-09037-0
eBook Packages: Computer ScienceComputer Science (R0)