Introduction: Brain-Computer Interfaces for Artistic Expression

Abstract

Capturing brain activity and translating it into multisensorial artistic expressions has been done by artists since the late sixties and early seventies of the previous century. At that time there were only very limited ways to acquire, process, manipulate and transform brain activity. No computing power, no pattern recognition, no machine learning, no graphics, no friendly user interfaces. The results of the transformations were usually presented visually, for example, on an oscilloscope, or auditorily using loudspeakers. In subsequent decades, brain-computer interfacing became a well-established research area that focused on applications in the clinical domain; in particular, applications that aimed at restoring and enhancing communication for the motor-impaired and for rehabilitation purposes. In more recent decades, due to progress in neuroscience, signal processing, and machine learning and progress in sensor technology, we see a growing interest in research and development that aims at clinical and nonclinical users that can use brain-computer interfaces for communication and control in real-life domestic, entertaining, and artistic brain-computer interfaces. This introductory chapter provides some general background on brain-computer interfaces. It mentions some standard paradigms, it provides some historical context and it presents some observations on brain-computer interfacing for artistic expression since the early seventies of the previous century. Currently, there is a market for inexpensive electroencephalographic (EEG) devices and software kits that capture voluntarily and involuntarily evoked brain activity and have this activity translated into control and communication commands for environments and devices. We also see a renewed interest of artists to make use of such devices to design interactive artistic installations that have knowledge of the brain activity of an individual user or the collective brain activity of a group of users, for example, an audience. This chapter provides some background on brain-computer interface technology that can be helpful for understanding the chapters that appear in this book and this chapter provides some context to the developments that are reported and foreseen in this book’s chapters.

Appendix

Nina Sobell, The Brain Streaming Project (2003): Proposal to the Rockefeller Foundation

In 2003 Nina Sobell submitted a proposal (N. Sobell. The Brain Streaming Project. Project proposal for the Rockefeller Foundation New Media Fellowships.) to the Rockefeller Foundation with a view on how Internet and the World Wide Web (WWW) could be used to communicate, share brain activity and engage with remote others to perform a collaborative painting task. Underlying the proposal was an earlier experiment. As mentioned on the website of the Brain Streaming Project (www.brainstreaming.org): “On May 18, 2002, our first transmission operated over the Internet for the first time, letting people in Manhattan and Brooklyn create a collaborative brain-wave drawing on the Web. By making Brainwave Drawings a Web event driven by custom client-server software, a new kind of genre of telepresence will begin to emerge in a multiple-node NetArt performance with accessible physical spaces.” The costs of this proposed ‘Brain Streaming Project’ were estimated to be $35,000. Unfortunately, it was not approved by the Rockefeller Foundation. The text of the main part of the proposal, slightly adapted, follows below.

Proposal

The Brain Streaming Project presents the means for people to connect and collaborate with one another by using only their brain waves. This non-verbal communication will be represented as a continuously evolving aural and visual expression, accessible to anyone logging on. The Brain Streaming Project will premiere with a 1-h international performance at physical and virtual locations including pocket computers and cellphones at www.brainstreaming.org. The transformed photo booth installations for the premiere performance will remain at each location for participation for the duration of the Fellowship term. For the duration of the performance, participants will be connected to electroencephalographs that amplify and identify their brain waves. The individual logon and brainwave data will be sent to the project server over the Internet, and entered into the server’s database. The server then streams this information to the project’s Web page, along with sounds and images that change dynamically as new input is received and viewed on touch screen monitors inside the booths, and on the Web. Brain Streaming is a metaphor for universal human consciousness. It reflects our similarities through the transformation of our converging thought patterns into the creation of a collaborative virtual collage.

My collaborators, M. E. Trivich, a systems engineer, Dr. John Dubberstein, a neurosurgeon and a musician, Sun Qing, a programmer and a theoretical physicist are committed to building our new form of universal language. Now that our custom client/server software has been successfully developed, we can devote our attention to discovering new ways of representing the EEG data into dynamically changing sounds and images, and designing a meaningful graphical user interface. Together we seek to create the meeting point at the intersection of art and technology.

Installation Plan: The Brain Streaming project will take place on pocket computers, cellphones, home computers and installed at three art/academic spaces located in three different cities. Three typical passport photo booths will be installed at the physical spaces. The exterior visual appearance of the booths will remain the same, the interior of the photo booths will appear much like their initial form but be equipped with web cams, monitors, and headgear with EEG (wireless electrodes) and headphones.

• A visitor approaches the photo booth with typical instructions posted outside that include extra features about the piece. They enter by pulling back the curtain, and adjusting their seat, so the web cam can see them. They put on the headgear equipped with EEG (wireless electrodes) and headphones.

• They enter the name of the city and the country they are from on the touch screen monitor facing them; longitude and latitude are automatically registered.

• They see their web cam image placed on their registered geographic location.

• They see the color of their faces change in accordance with the output of their own brain waves, and hear the output of their brain waves through the headphones.

• On the same login screen, they touch a button, which brings them to a collaborative painting and composing site.

• Then they hear and see the transformation of their brain wave output mixed with other participants. Volume as dynamics; Beta as rhythm; Alpha as tempo; Theta as pitch, and other parameters expressing the complexity of harmonics.

• The installation will remain open during gallery hours, and 24/7 for those with the headgear on the web at home, cellphones or pocket computers with built-in webcams, and those who logon to observe.

• Those who logon to observe will be identified by a color they choose from a color wheel.

In this interface, three typical passport booths in three cities are represented by the yellow circles. The yellow circles are placed at locations where photo booths will be hooked to EEGs. Surrounding them are expanding and contracting circles which illustrate amplitude through size and brain wave types and color. Red squares indicate the locations of visitors. Dimming and brightening of the map is a rough illustration of one way to arrogate the data from the EEG participants. In this illustration, when the circle is brighter, it is intended to illustrate when their brain waves are in sync. This diagrammatic page leads both physical and virtual participants to a full screen painting and music composition they are creating together, as illustrated in the model below.

Brain Streaming Project 2003