Skip to main content
SpringerLink
Log in

Large GAN Is All You Need

  • Conference paper
  • First Online:
Advances in Computer Graphics (CGI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14495))

Included in the following conference series:

  • 194 Accesses

Abstract

Sketch-to-portrait conversion is an emerging research area that aims to transform rough facial line sketches into highly detailed and realistic portrait images. This paper presents a comprehensive study on the impact of different loss functions and data augmentation techniques in achieving superior results using the U-Net256 network architecture. The study explores the effects of Mean Squared Error (MSE) loss, L1 loss, Generative Adversarial Network (GAN) loss, and the number of parameters on the quality of the generated portrait images. Experimental results demonstrate that the choice of loss function significantly influences the perceptual quality and accuracy of the converted portraits. While both MSE and L1 loss contribute to capturing the overall structure, GAN loss excels in generating fine-grained details. Moreover, a trade-off is observed between the number of parameters and image quality, with higher parameter counts resulting in more intricate outputs but increased computational complexity. In conclusion, this paper offers valuable insights into the sketch to portrait conversion task, shedding light on the effects of different loss functions and data augmentation techniques. The findings contribute to the advancement of sketch-to-portrait conversion systems, pushing the boundaries of realism and detail in generated portrait images. We finally reached FID value of 0.2184, the fourth in the CGI-PSG2023 leaderboard as of September 21st. The dataset can be found on CGI-PSG2023 webpage. All code is open-source and can be found in https://github.com/KKK-Liu/Portrait.git

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Choi, Y., Choi, M., Kim, M., Ha, J., Kim, S., Choo, J.: StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation (2017). CoRR abs/1711.09020, http://arxiv.org/abs/1711.09020

  2. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016). https://doi.org/10.1109/CVPR.2016.265

  3. Goodfellow, I.J., et al.: Generative adversarial networks (2014)

    Google Scholar 

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)

    Google Scholar 

  5. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium (2018)

    Google Scholar 

  6. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift (2015)

    Google Scholar 

  7. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976 (2017). https://doi.org/10.1109/CVPR.2017.632

  8. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks (2018)

    Google Scholar 

  9. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017)

    Google Scholar 

  10. Lata, K., Dave, M., Nishanth, K.N.: Image-to-image translation using generative adversarial network. In: 2019 3rd International conference on Electronics, Communication and Aerospace Technology (ICECA), pp. 186–189 (2019). https://doi.org/10.1109/ICECA.2019.8822195

  11. Li, P., Sheng, B., Chen, C.L.P.: Face sketch synthesis using regularized broad learning system. IEEE Trans. Neural Netw. Learn. Syst. 33(10), 5346–5360 (2022). https://doi.org/10.1109/TNNLS.2021.3070463

    Article  Google Scholar 

  12. Li, S., Wu, F., fan, Y., Song, X., Dong, W.: PLDGAN: portrait line drawing generation with prior knowledge and conditioning target. Vis. Comput. 39(8), 3507–3518 (2023). https://doi.org/10.1007/s00371-023-02956-1

  13. Li, Z., Togo, R., Ogawa, T., Haseyama, M.: Semantic-aware unpaired image-to-image translation for urban scene images. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2150–2154 (2021). https://doi.org/10.1109/ICASSP39728.2021.9414192

  14. Liu, M.Y., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., Kautz, J.: Few-shot unsupervised image-to-image translation (2019)

    Google Scholar 

  15. Nozawa, N., Shum, H.P.H., Feng, Q., Ho, E.S.L., Morishima, S.: 3D car shape reconstruction from a contour sketch using GAN and lazy learning. Vis. Comput. 38(4), 1317–1330 (2022). https://doi.org/10.1007/s00371-020-02024-y

  16. Park, T., Liu, M., Wang, T., Zhu, J.: Semantic image synthesis with spatially-adaptive normalization (2019). CoRR abs/1903.07291, http://arxiv.org/abs/1903.07291

  17. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation (2015)

    Google Scholar 

  18. Tschannen, M., Agustsson, E., Lucic, M.: Deep generative models for distribution-preserving lossy compression (2018)

    Google Scholar 

  19. Tsuda, H., Hotta, K.: Cell image segmentation by integrating pix2pixs for each class. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1065–1073 (2019). https://doi.org/10.1109/CVPRW.2019.00139

  20. Uzunova, H., Ehrhardt, J., Jacob, F., Frydrychowicz, A., Handels, H.: Multi-scale GANs for memory-efficient generation of high resolution medical images (2019)

    Google Scholar 

  21. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  22. Yeh, R.A., Chen, C., Lim, T.Y., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N.: Semantic image inpainting with deep generative models (2017)

    Google Scholar 

  23. Yoshikawa, T., Endo, Y., Kanamori, Y.: Diversifying detail and appearance in sketch-based face image synthesis. The Visual Computer (Proc. of Computer Graphics Internatinal 2022) (2022)

    Google Scholar 

  24. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Shanghai Jiao Tong University, Shanghai, 200240, China

    Kai Liu, Qingyang Wu & Mengkun Xie

Authors
  1. Kai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingyang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mengkun Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kai Liu .

Editor information

Editors and Affiliations

  1. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  2. Shanghai Jiao Tong University, Shanghai, China

    Lei Bi

  3. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  4. MIRALab-CUI, University of Geneve, Carouge, Geneve, Switzerland

    Nadia Magnenat-Thalmann

  5. Swiss Federal Institute of Technology, Lausanne, Switzerland

    Daniel Thalmann

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, K., Wu, Q., Xie, M. (2024). Large GAN Is All You Need. In: Sheng, B., Bi, L., Kim, J., Magnenat-Thalmann, N., Thalmann, D. (eds) Advances in Computer Graphics. CGI 2023. Lecture Notes in Computer Science, vol 14495. Springer, Cham. https://doi.org/10.1007/978-3-031-50069-5_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50069-5_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50068-8

  • Online ISBN: 978-3-031-50069-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation