Skip to main content
SpringerLink
Log in

ImaGAN: Unsupervised Training of Conditional Joint CycleGAN for Transferring Style with Core Structures in Content Preserved

  • Conference paper
  • First Online:
Book cover Computer Vision – ACCV 2018 (ACCV 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11362))

Included in the following conference series:

  • 2168 Accesses

Abstract

This paper considers conditional image generation that merges the structure of one object with the style of another. In short, the style of an image has been substituted or replaced by the style of another image. An architecture for extracting the structure of one image and another architecture for merging the extracted structure and the style of another image is considered. The proposed ImaGAN architecture consists of two networks: (1) style removal network R that removes style information and leaves only the edge information and (2) the generation network G that fills the extracted structure with the style of another image. This architecture allows various pairing of style and structure which would not have been possible with image-to-image translation method. This architecture incurs minimal classification error compared prior style transfer and domain transfer algorithms. Experimental result using edges2handbags and edges2shoes dataset reveal that ImaGAN can transfer the style of one image to another without distorting the target structure.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Azadi, S., Fisher, M., Kim, V.G., Wang, Z., Shechtman, E., Darrell, T.: Multi-content GAN for few-shot font style transfer. CoRR abs/1712.00516 (2017). http://arxiv.org/abs/1712.00516

  2. Chang, H., Lu, J., Yu, F., Finkelstein, A.: PairedCycleGAN: asymmetric style transfer for applying and removing makeup. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  3. Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: StyleBank: an explicit representation for neural image style transfer. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 2770–2779 (2017)

    Google Scholar 

  4. Chen, T.Q., Schmidt, M.: Fast patch-based style transfer of arbitrary style. CoRR abs/1612.04337 (2016). http://arxiv.org/abs/1612.04337

  5. Deng, J., Berg, A., Satheesh, S., Su, H., Khosla, A., Fei-Fei, L.: ILSVRC 2012 (2012). http://www.image-net.org/challenges/LSVRC

  6. Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. CoRR abs/1610.07629 (2016). http://arxiv.org/abs/1610.07629

  7. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016, pp. 2414–2423 (2016)

    Google Scholar 

  8. Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014). http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf

  9. Gupta, A., Johnson, J., Alahi, A., Fei-Fei, L.: Characterizing and improving stability in neural style transfer. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22–29 October 2017, pp. 4087–4096 (2017)

    Google Scholar 

  10. Huang, X., Belongie, S.J.: Arbitrary style transfer in real-time with adaptive instance normalization. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22–29 October 2017, pp. 1510–1519 (2017)

    Google Scholar 

  11. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 5967–5976 (2017)

    Google Scholar 

  12. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part II. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  13. Joo, D., Kim, D., Kim, J.: Generating a fusion image: one’s identity and another’s shape. CoRR abs/1804.07455 (2018). http://arxiv.org/abs/1804.07455

  14. Kaur, P., Zhang, H., Dana, K.J.: Photo-realistic facial texture transfer. CoRR abs/1706.04306 (2017). http://arxiv.org/abs/1706.04306

  15. Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017. Proceedings of Machine Learning Research, vol. 70, pp. 1857–1865. PMLR, International Convention Centre (2017). http://proceedings.mlr.press/v70/kim17a.html

  16. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980

  17. Li, C., Wand, M.: Combining Markov random fields and convolutional neural networks for image synthesis. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 2479–2486 (2016)

    Google Scholar 

  18. Li, Y., Wang, N., Liu, J., Hou, X.: Demystifying neural style transfer. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, 19–25 August 2017, pp. 2230–2236 (2017)

    Google Scholar 

  19. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.: Universal style transfer via feature transforms. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017, pp. 385–395 (2017). http://papers.nips.cc/paper/6642-universal-style-transfer-via-feature-transforms

  20. Maini, R., Aggarwal, H.: Study and comparison of various image edge detection techniques

    Google Scholar 

  21. Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS-W (2017)

    Google Scholar 

  22. Shen, F., Yan, S., Zeng, G.: Meta networks for neural style transfer. CoRR abs/1709.04111 (2017). http://arxiv.org/abs/1709.04111

  23. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  24. Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.S.: Texture networks: feed-forward synthesis of textures and stylized images. In: Proceedings of the 33rd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, 19–24 June 2016, pp. 1349–1357 (2016). http://jmlr.org/proceedings/papers/v48/ulyanov16.html

  25. Wilmot, P., Risser, E., Barnes, C.: Stable and controllable neural texture synthesis and style transfer using histogram losses. CoRR abs/1701.08893 (2017). http://arxiv.org/abs/1701.08893

  26. Xian, W., et al.: TextureGAN: controlling deep image synthesis with texture patches. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  27. Yu, A., Grauman, K.: Fine-grained visual comparisons with local learning. In: Computer Vision and Pattern Recognition (CVPR), June 2014

    Google Scholar 

  28. Zhu, J.-Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part V. LNCS, vol. 9909, pp. 597–613. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_36

    Chapter  Google Scholar 

  29. Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

Download references

Acknowledgments

This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT & Future Planning(NRF-2017R1A2B2006165) and Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) (No. 2018-0-00198), Object information extraction and real-to-virtual mapping based AR technology.

Author information

Authors and Affiliations

  1. KAIST, Daejeon, South Korea

    Kang Min Bae, Minuk Ma, Hyunjun Jang, Minjeong Ju, Hyoungwoo Park & Chang D. Yoo

Authors
  1. Kang Min Bae
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minuk Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hyunjun Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Minjeong Ju
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hyoungwoo Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chang D. Yoo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kang Min Bae .

Editor information

Editors and Affiliations

  1. IIIT Hyderabad, Hyderabad, India

    C. V. Jawahar

  2. ANU, Canberra, ACT, Australia

    Hongdong Li

  3. Simon Fraser University, Burnaby, BC, Canada

    Greg Mori

  4. ETH Zurich, Zurich, Zürich, Switzerland

    Konrad Schindler

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bae, K.M., Ma, M., Jang, H., Ju, M., Park, H., Yoo, C.D. (2019). ImaGAN: Unsupervised Training of Conditional Joint CycleGAN for Transferring Style with Core Structures in Content Preserved. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11362. Springer, Cham. https://doi.org/10.1007/978-3-030-20890-5_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20890-5_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20889-9

  • Online ISBN: 978-3-030-20890-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation