Abstract
In this paper we review the overall process for the design, development, and deployment of “What I See Is What You Get”, an experiential installation that creates live interactive visuals, by analyzing human facial expressions and behaviors, accompanied by text generated using Machine Learning algorithms trained on the art collection of The J. Paul Getty Museum in Los Angeles. The project is developed by students and faculty in an academic environment and exhibited at the Getty Museum. We also study the pedagogical process implemented to address the curriculum’s learning outcomes in an “applied” environment while designing a contemporary new media art piece. Special attention is paid to the level and quality of the interaction between users and the piece, demonstrating how advances in technology and computing such as Deep Learning and Natural Language Processing can contribute to deeper connections and new layers of interactivity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tribe, M., Jana, R., Grosenick, U.: New Media Art (Taschen Basic Art). Taschen America, LLC (2006)
Krueger, M.W.: Responsive environments. In: AFIPS 1977 Proceedings of the June 13–16, 1977, National Computer Conference, pp. 423–433. Dallas, Texas (1977)
Krauth AK: Using handmade controllers for interactive projection mapping. In: Proceedings of the 23rd ACM International Conference on Multimedia - MM 2015, pp 717–719. ACM Press, Brisbane, Australia (2015)
Manovich, L.: Automating aesthetics: artificial intelligence and image culture. In: Flash Art International, no. 316 (2017). https://flash---art.com/issue/316-september-october-2017/. Accessed 14 Aug 2019
OpenCV. https://opencv.org/. Accessed 10 Aug 2019
PyCharm: The Python IDE for Professional Developers by JetBrains. https://www.jetbrains.com/pycharm/. Accessed 10 Aug 2019
Derivative TouchDesigner. https://www.derivative.ca/. Accessed 10 Aug 2019
KINECTONE. https://developer.microsoft.com/en-us/windows/kinect. Accessed 10 Aug 2019
Structure Sensor-3D scanning, augmented reality, and more for mobile devices. https://structure.io/. Accessed 10 Aug 2019
Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004). Kluwer Academic Publishers Norwell, MA
Teixeira Lopes, A., de Aguiar, E., Oliveira-Santos, T.: A facial expression recognition system using convolutional networks. In: Conference on Graphics, Patterns and Images (SIBGRAPI), 28th Conference, Salvador, Brazil, pp. 273–280 (2015)
Rothe, R., Timofte, R., Gool, L.V.: DEX: deep expectation of apparent age from a single image. In: 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), pp. 252–257 (2015)
Keras Documentation. https://keras.io/. Accessed 10 Aug 2019
TensorFlow. https://www.tensorflow.org/. Accessed 10 Aug 2019
Radford, A., Wu, J., Child, R., Luan, D., Amode, D., Sutskever, I.: Language models are unsupervised multitask learners. In: Technical report, OpenAI (2019). https://openai.com/blog/better-language-models/. Accessed 14 Aug 2019
Google Colaboratory. https://github.com/ak9250/gpt-2-colab. Accessed 10 Oct 2019
Python Data Analysis Library. https://pandas.pydata.org/. Accessed 18 Oct 2019
Woolf, M.: Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code: minimaxir/textgenrnn (2015). https://github.com/minimaxir/textgenrn
Kaggle Facial Expression Recognition Challenge. https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/. Accessed 18 Oct 2019
ACS-Woodbury Github Repository. https://github.com/ACS-Woodbury/WISIWYG.git. Accessed 19 Oct 2019
Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015). http://www.robots.ox.ac.uk/~vgg/publications/2015/Parkhi15/parkhi15.pdf
Large Scale Visual Recognition Challenge 2014. http://image-net.org/challenges/LSVRC/2014/. Accessed 18 Oct 2019
IMDB-WIKI Dataset - 500 k+ face images with age and gender labels. https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/. Accessed 10 Aug 2019
Apparent Age and Gender Prediction in Keras. https://sefiks.com/2019/02/13/apparent-age-and-gender-prediction-in-keras/
Connor, A.M., Sosa, R.: The A-Z of creative technologies. EAI Endorsed Trans. Creative Technol. 5, 1–2 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Herruzo, A., Pashenkov, N. (2020). “What I See Is What You Get” Explorations of Live Artwork Generation, Artificial Intelligence, and Human Interaction in a Pedagogical Environment. In: Brooks, A., Brooks, E. (eds) Interactivity, Game Creation, Design, Learning, and Innovation. ArtsIT DLI 2019 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 328. Springer, Cham. https://doi.org/10.1007/978-3-030-53294-9_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-53294-9_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-53293-2
Online ISBN: 978-3-030-53294-9
eBook Packages: Computer ScienceComputer Science (R0)