Skip to main content
SpringerLink
Log in

Interactive, Efficient and Creative Image Generation Using Compositional Pattern-Producing Networks

  • Conference paper
  • First Online:
Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12693))

Abstract

In contrast to most recent models that generate an entire image at once, the paper introduces a new architecture for generating images one pixel at a time using a Compositional Pattern-Producing Network (CPPN) as the generator part in a Generative Adversarial Network (GAN), allowing for effective generation of visually interesting images with artistic value, at arbitrary resolutions independent of the dimensions of the training data. The architecture, as well as accompanying (hyper-) parameters, for training CPPNs using recent GAN stabilisation techniques is shown to generalise well across many standard datasets. Rather than relying on just a latent noise vector (entangling various features with each other), mutual information maximisation is utilised to get disentangled representations, removing the requirement to use labelled data and giving the user control over the generated images. A web application for interacting with pre-trained models was also created, unique in the offered level of interactivity with an image-generating GAN.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The supplement gives further example outputs: https://bit.ly/impgan_sup.

  2. 2.

    https://github.com/zo7/painter-by-numbers/releases/tag/data-v1.0.

  3. 3.

    https://vuejs.org/ resp. https://www.tensorflow.org/js.

  4. 4.

    See the supplement for visuals from these baseline experiments, further example outputs at different resolutions and hyper-parameter settings, and videos displaying super-resolution effects and the interactive application: https://bit.ly/impgan_sup.

  5. 5.

    As given on http://github.com/NVlabs/ in stylegan resp. stylegan2.

References

  1. Anokhin, I., Demochkin, K., Khakhulin, T., Sterkin, G., Lempitsky, V., Korzhenkov, D.: Image generators with conditionally-independent pixel synthesis. CoRR abs/2011.13775, November 2020. https://arxiv.org/abs/2011.13775

  2. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. CoRR abs/1701.07875, December 2017. https://arxiv.org/abs/1701.07875

  3. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. CoRR abs/1607.06450, July 2016. https://arxiv.org/abs/1607.06450

  4. Boden, M.A.: The Creative Mind: Myths and Mechanisms. 2 edn. Routledge (2003)

    Google Scholar 

  5. Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: 7th International Conference on Learning Representations, New Orleans, LA, USA, May 2019. https://arxiv.org/abs/1809.11096

  6. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: InfoGAN: interpretable representation learning by information maximizing generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 30, pp. 2172–2180. NIPS, Barcelona, Spain, December 2016

    Google Scholar 

  7. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 249–256. PMLR, Chia Laguna, Italy, May 2010

    Google Scholar 

  8. Goodfellow, I., et al.: Generative adversarial networks. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. NIPS, Montréal, Canada, December 2014. https://arxiv.org/abs/1406.2661

  9. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems, vol. 30, pp. 5767–5777. NIPS, Long Beach, CA, USA, December 2017

    Google Scholar 

  10. Ha, D.: Neural network generative art in Javascript, Jun 2015. http://blog.otoro.net/2015/06/19/neural-network-generative-art/

  11. Ha, D.: Neurogram, July 2015. http://blog.otoro.net/2015/07/31/neurogram/

  12. Ha, D.: The frog of CIFAR 10, April 2016. http://blog.otoro.net/2016/04/06/the-frog-of-cifar-10/

  13. Ha, D.: Generating abstract patterns with TensorFlow, March 2016. http://blog.otoro.net/2016/03/25/generating-abstract-patterns-with-tensorflow/

  14. Ha, D.: Generating large images from latent vectors, April 2016. http://blog.otoro.net/2016/04/01/generating-large-images-from-latent-vectors/

  15. Ha, D.: Generating large images from latent vectors - part two, June 2016. http://blog.otoro.net/2016/06/02/generating-large-images-from-latent-vectors-part-two/

  16. Ha, D.: Interactive abstract pattern generation Javascript demo, April 2016. http://blog.otoro.net/2016/04/24/interactive-abstract-pattern-generation-javascript-demo/

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. IEEE, Las Vegas, NV, USA, June 2016

    Google Scholar 

  18. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 31, pp. 6626–6637. NIPS, Long Beach, CA, USA, December 2017

    Google Scholar 

  19. Ioffe, S.: Batch renormalization: towards reducing minibatch dependence in batch-normalized models. In: Advances in Neural Information Processing Systems, vol. 31, pp. 1945–1953. NIPS, Long Beach, CA, USA, December 2017

    Google Scholar 

  20. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 448–456. PMLR, Lille, France, June 2015

    Google Scholar 

  21. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4396–4405. IEEE, Long Beach, June 2019

    Google Scholar 

  22. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. CoRR abs/1912.04958, December 2019. https://arxiv.org/abs/1912.04958

  23. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980, January 2017. https://arxiv.org/abs/1412.6980

  24. Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto, April 2009

    Google Scholar 

  25. Kurach, K., Lučić, M., Zhai, X., Michalski, M., Gelly, S.: A large-scale study on regularization and normalization in GANs. In: Proceedings of the 36th International Conference on Machine Learning, pp. 3581–3590. PMLR, Long Beach, CA, USA, June 2019

    Google Scholar 

  26. LeCun, Y., Cortes, C., Burges, C.J.: The MNIST database of handwritten digits (2010). http://yann.lecun.com/exdb/mnist

  27. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of 2015 IEEE International Conference on Computer Vision, pp. 3730–3738. IEEE, Santiago, Chile, December 2015

    Google Scholar 

  28. Metz, L., Gulrajani, I.: Compositional pattern producing GAN. In: Workshop on Machine Learning for Creativity and Design, Long Beach, CA, USA, December 2017

    Google Scholar 

  29. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: 6th International Conference on Learning Representations, Vancouver, Canada, May 2018. https://arxiv.org/abs/1802.05957

  30. Odena, A.: Open questions about generative adversarial networks. Distill, April 2019. https://distill.pub/2019/gan-open-problems

  31. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, vol. 30, pp. 2234–2242. NIPS, Barcelona, Spain, December 2016

    Google Scholar 

  32. Secretan, J., et al.: Picbreeder: a case study in collaborative evolutionary exploration of design space. Evol. Comput. 19(3), 373–403 (2011)

    Article  Google Scholar 

  33. Sims, K.: Artificial evolution for computer graphics. Comput. Graph. 25(4), 319–328 (1991)

    Article  Google Scholar 

  34. Stanley, K.O.: Compositional pattern producing networks: a novel abstraction of development. Genet. Program. Evolvable Mach. 8(2), 131–162 (2007)

    Article  Google Scholar 

  35. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  36. The TensorFlow Team: Flowers, January 2019. http://download.tensorflow.org/example_images/flower_photos.tgz

  37. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747, September 2017. https://arxiv.org/abs/1708.07747

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Norwegian University of Science and Technology, 7491, Trondheim, Norway

    Erlend Gjesteland Ekern & Björn Gambäck

  2. Digital Systems, RISE, Research Institute of Sweden AB, 164 29, Kista, Sweden

    Björn Gambäck

Authors
  1. Erlend Gjesteland Ekern
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Björn Gambäck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Björn Gambäck .

Editor information

Editors and Affiliations

  1. University of A Coruña, A Coruña, Spain

    Juan Romero

  2. University of Coimbra, Coimbra, Portugal

    Tiago Martins

  3. University of A Coruña, A Coruña, Spain

    Nereida Rodríguez-Fernández

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ekern, E.G., Gambäck, B. (2021). Interactive, Efficient and Creative Image Generation Using Compositional Pattern-Producing Networks. In: Romero, J., Martins, T., Rodríguez-Fernández, N. (eds) Artificial Intelligence in Music, Sound, Art and Design. EvoMUSART 2021. Lecture Notes in Computer Science(), vol 12693. Springer, Cham. https://doi.org/10.1007/978-3-030-72914-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72914-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72913-4

  • Online ISBN: 978-3-030-72914-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation