Pan Li
ABSTRACT
Image style transfer is a classic image editing task which aims to transfer arbitrary visual styles to content images. In recent years, it has been revealed that a well-trained convolutional neural network with sufficient labeled data is powerful to deal with the style transfer problem. Thanks to the recent advances in the analysis of neural style transfer, the image style transfer can be cast as a problem of distribution alignment. In this paper, we propose to solve this issue by incorporating the theory of optimal transport in a simple and intuitive way. The main component of our style transfer method is an optimal transportation map, which is derived from the Monge-Kantorovicth theory of mass transportation, to perform the alignment process from the content image to the style image. We compare the generated stylized images with a number of representative algorithms to demonstrate the effectiveness of our approach. We also show that our results are visually more consistent and well-stylized simultaneously.
- Ashikhmin, N. (2003). Fast texture transfer. Computer Graphics & Applications IEEE, 23(4), 38--43. Google Scholar
Digital Library
- Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2414--2423).Google ScholarCross Ref
- Gatys, L. A., Ecker, A. S., & Bethge, M. (2015). A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576.Google Scholar
- Gooch, B., & Gooch, A. (2001). Non-photorealistic rendering. AK Peters/CRC Press. Google ScholarDigital Library
- Strothotte, T., & Schlechtweg, S. (2002). Non-photorealistic computer graphics: modeling, rendering, and animation. Morgan Kaufmann. Google ScholarDigital Library
- Lee, H., Seo, S., Ryoo, S., & Yoon, K. (2010, June). Directional texture transfer. In Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering (pp. 43--48). ACM. Google ScholarDigital Library
- Efros, A. A., & Leung, T. K. (1999, September). Texture synthesis by non-parametric sampling. In iccv (pp. 1033). IEEE. Google ScholarDigital Library
- Gatys, L., Ecker, A. S., & Bethge, M. (2015). Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems (pp. 262--270). Google ScholarDigital Library
- Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097--1105). Google ScholarDigital Library
- Efros, A. A., & Freeman, W. T. (2001, August). Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (pp. 341--346). ACM. Google ScholarDigital Library
- Frigo, O., Sabater, N., Delon, J., & Hellier, P. (2016). Split and match: Example-based adaptive patch sampling for unsupervised style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 553--561).Google ScholarCross Ref
- Elad, M., & Milanfar, P. (2017). Style Transfer via Texture Synthesis. IEEE Trans. Image Processing, 26(5), 2338--2351. Google ScholarDigital Library
- Li, C., & Wand, M. (2016). Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2479--2486).Google ScholarCross Ref
- Johnson, J., Alahi, A., & Fei-Fei, L. (2016, October). Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (pp. 694--711). Springer, Cham.Google Scholar
- Ulyanov, D., Lebedev, V., Vedaldi, A., & Lempitsky, V. S. (2016, June). Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. In ICML (pp. 1349--1357). Google ScholarDigital Library
- Portilla, J., & Simoncelli, E. P. (2000). A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40(1), 49--70. Google ScholarDigital Library
- Gatys, L. A., Ecker, A. S., Bethge, M., Hertzmann, A., & Shechtman, E. (2017, July). Controlling perceptual factors in neural style transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Dumoulin, V., Shlens, J., & Kudlur, M. (2017). A learned representation for artistic style. Proc. of ICLR.Google Scholar
- Chen, D., Yuan, L., Liao, J., Yu, N., & Hua, G. (2017, July). Stylebank: An explicit representation for neural image style transfer. In Proc. CVPR (Vol. 1, No. 3, p. 4).Google Scholar
- Li, Y., Wang, N., Liu, J., & Hou, X. (2017). Demystifying neural style transfer. arXiv preprint arXiv:1701.01036. Google ScholarDigital Library
- Courty, N., Flamary, R., Tuia, D., & Rakotomamonjy, A. (2017). Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9), 1853--1865.Google ScholarDigital Library
- Courty, N., Flamary, R., & Tuia, D. (2014, September). Domain adaptation with regularized optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 274--289). Springer, Berlin, Heidelberg. Google ScholarDigital Library
- Chen, T. Q., & Schmidt, M. (2016). Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv:1612.04337.Google Scholar
- Huang, X., & Belongie, S. J. (2017, February). Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. In ICCV (pp. 1510--1519).Google Scholar
- Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., & Yang, M. H. (2017). Universal style transfer via feature transforms. In Advances in Neural Information Processing Systems (pp. 386--396). Google ScholarDigital Library
- Monge, G. (1781). Mémoire sur la théorie des déblais et des remblais. Histoire de l'Académie Royale des Sciences de Paris.Google Scholar
- Kantorovitch, L. (1958). On the translocation of masses. Management Science, 5(1), 1--4. Google ScholarDigital Library
- Knott, M., & Smith, C. S. (1984). On the optimal mapping of distributions. Journal of Optimization Theory and Applications, 43(1), 39--49.Google ScholarCross Ref
- K. Nichol. Painter by numbers, wikiart. https:// www.kaggle.com/c/painter-by-numbers, 2016.Google Scholar
- Perrot, M., Courty, N., Flamary, R., & Habrard, A. (2016). Mapping estimation for discrete optimal transport. In Advances in Neural Information Processing Systems (pp. 4197--4205) Google ScholarDigital Library
Index Terms
- Optimal Transport of Deep Feature for Image Style Transfer
Recommendations
Aesthetic-Aware Image Style Transfer
MM '20: Proceedings of the 28th ACM International Conference on MultimediaStyle transfer aims to synthesize an image which inherits the content of one image while preserving a similar style of the other one. The "style'' of an image usually refers to its unique feeling conveyed from visual features, which is highly related to ...
Arbitrary Image Emotionalizing with Style Transfer
Advances in Multimedia Information Processing – PCM 2018AbstractEndowing user-specified emotions on arbitrary images will be an interesting application. The emotion transfer of landscape photograph has shown greater appeal than the emotion transfer of facial image in the field of art design or painting. In ...
Soft multimodal style transfer via optimal transport
AbstractThe goal of arbitrary Neural Style Transfer (NST) is to transform a content image into a different style. Based on the assumption that styles can be represented by global statistics, the majority of NST methods seek to match global ...
Comments