Keywords

1 Introduction

There is a long tradition of using brain signals in artistic practice. Since Lucier’s Music for Solo Performer [12], there have been further explorations of these signals in music, visual arts and performance. In parallel, in the last forty years, practice as research (PaR) has become an accepted methodology for research in the arts. A key characteristic of PaR is that it deals with embodied knowledge, knowledge that is located in the body and cannot be reduced to a set of quantities or a discursive text. Nevertheless, there is a need for descriptions about practice both for critical reflection and to evidence the research inquiry [16]. During our work with an artist with Locked-In Syndrome, we have identified a possible enhancement upon the kinds of available descriptions. Following Nelson [17], we think that these descriptions should be multi-modal and, as embodied knowledge is central to PaR, we propose that the methodology may benefit from tools that provide information about the body. This paper presents a framework for using BCI in PaR and it is organized as follows: Sect. 2 is a short explanation of PaR and its process, Sect. 3 describes our work, the documentation possibilities we have identified and presents the concept of body queries as a way to obtain and interpret data from the body, Sect. 4 explains the framework for using BCI in practice as research projects, Sect. 5 provides a review of three examples of BCI in artistic or art-related to illustrate the framework, and Sect. 6 presents the discussion.

2 Practice as Research

In his 2006 manifesto, Haseman [7] made the case for the formalization of a new paradigm of research, different from quantitative and qualitative, that he called performative research. In this paradigm, practice is central to the research process and its findings are presented as symbolic data in the particular forms of the practice. He argued that, while methods from the other paradigms might be adapted and used, practitioner-researchersFootnote 1 can also invent their own methods to inquire upon their practices. This was exemplified by the method of artistic audit, where the focus is placed on attending the symbolic forms of the artwork in performance while placing them in the context of the practice and the research inquiry. An artistic audit of a painting, for example, implies going beyond the mere act of gazing by placing the information gathered by the gaze in the traditions and conventions that contextualize the piece. Later, Nelson [17] encoded information from past experiences using the paradigm and proposed a multi-mode epistemological model to support practice as research (PaR), which he defined as involving projects

in which practice is a key method of inquiry and where, in respect of the arts, a practice (creative writing, dance, musical score/performance, theatre/performance, visual exhibition, film or other cultural practice) is submitted as substantial evidence of a research inquiry.

Nelson’s model (Fig. 1) articulates three modes of knowledge in a loop, modes that resonate between them and across arts praxis, or “theory imbricated with practice”. These modes of knowledge are: (i) know-how, located in the body, experiential and tacit; (ii) know-what, the result of critical reflection aiming to make the tacit explicit; and (iii) know-that, explicit, shared, articulated socially, and propositional.

Fig. 1.
figure 1

Simplified version of Nelson’s multi-mode epistemological model. Each mode of knowing resonates with the others in order to “articulate the tacit” inherent in practice. The original is found in [16].

Full size image

The model codifies the idea that knowledge exists on a spectrum that goes from tacit (embodied, performative, semi-conscious or unconscious) to explicit (objective or maximally intersubjective, susceptible of representation by numbers or words), and that practitioner-researchers should strive to find processes that articulate the tacit even if cannot be made thoroughly explicit. To that end, Nelson [15] recommends that “if practitioner-researchers wish their embodied cognitions to be better recognised, means of identifying and disseminating them must be sought”.

Complementary documentation and writing are the means to make the tacit explicit. Their goal is to draw attention to the doing-thinking of the practitioner-researcher during the process. They are thick descriptions, that is, descriptions that explain the activity and the broader context which makes it meaningful in order to provide a better understanding of it. By working on these documents, practitioner-researchers engage in critical reflection about their practice, moving from know-how to know-what and finding resonances to know-that. As a result, their practice is enhanced and the loop begins anew.

3 Body Queries

We have been working with an actor and playwright with Locked-In SyndromeFootnote 2 since 2009. We developed a custom-built communication system based on an eye-tracking device to support him both in daily life and in his artistic practice. He has written a play, a book and has participated in several research teams. He is working on a new play and collaborating with our team in the design and implementation of a brain-computer music interface, in the context of finding ways to implement BCI-based systems for the performing arts [1].

During our work, and due to his condition, we have faced an increasing need for information about the body state of the artist during practice. As documentation of process in PaR is a multi-modal affair, its sources can be words, diagrams, drawings, sketches, video and sound recordings of rehearsals, material objects produced during practice, or printed material related to the activity [18]. However, these sources are once or twice removed from the body that performs the activity. By this we mean that they gather information about the body from the results of the activity, or by the recording of those results. The closest to directly asking the body comes when expert practitioner-researchers are able to recognize changes in their bodies as they perform and reflect upon them afterwards.

Documentation is mostly created and analyzed by the practitioner-researchers themselves. In our case, this is not possible with the usual methods. While there are methods to guide the design of BCI tools for users with motor impairments (see [9, 11] for examples), they are focused on the design of the BCI system itself and not in its role in a PaR process.

Keeping in mind that the goal is to make the tacit explicit, we think there is a documentation gap between first-person experience and currently used methods. This gap could be filled by technological tools that query the body directly and provide information about its state and its changes during practice. This information, in turn, can be interpreted according to the requirements of the practice and the know-that that supports the tool being used. We propose the term body query to name this process of obtaining information about the body using technology and interpreting that information according to previously agreed-upon criteria (Fig. 2).

Fig. 2.
figure 2

The body query process. Bodily source information is gathered by the query tool, this information is interpreted according to criteria related to practice and the specific query tool, and this interpretation is compiled as documentation.

Full size image

3.1 Bodily Sources

Any signal produced by the body that can be gathered with a technological device is a potential source for body queries. These signals must have, at least, a previously agreed-upon interpretation related to the query tool being used. That is, there should be a consensus about what kind of information the signal obtained with the device provides about the body and its state. Thus, there must be a context upon which the information from the signal can be interpreted. We place this constraint upon the source to guarantee the contextualization of the documentation in the know-that. It also works as a safeguard against reductionist interpretations by forcing reflection upon the kinds of claims a practitioner-researcher can make about the information gathered from the signal.

For example, writing about the trend of linking neuroscience and performance studies, May [13] proposes that practitioner-researchers should be aware that they can make four kinds of claim about the relationship between brain and practice:

  1. 1.

    Activation in brain area(s) X is correlated with Y.

  2. 2.

    Activation in brain area(s) X is necessary for Y to occur.

  3. 3.

    Activation in brain area(s) X is necessary and sufficient for Y to occur.

  4. 4.

    Y is actually, or is reducible to, activation in brain area(s) X.Footnote 3

In order to make one of these claims in a body query, a practitioner-researcher must be aware of the burden of proof their choice entails. As they are listed in increasing order of difficulty to prove, reducing one aspect of a practice to activation in one area of the brain is more difficult to support than saying that the aspect and the activation are correlated. In general, while executing body queries care must be exercised in what kind of interpretative claims are made and whether they are supported by existing evidence.

The signals may also be interpreted according to criteria based on the practice, whether extracted from previous uses in other projects or built during the course of practice. In this case, interpretation is more open in the sense that, while the criteria may change during the process, those changes should be adequately contextualized in the broader context of the research inquiry. This constraint should enforce the requirement of critical reflection upon the process while keeping interpretation flexible enough to adapt to the dynamics of the practice.

3.2 Query Tools

Any technological device that can gather signals from the body is a query tool. This implies that devices originally created for purposes other than artistic expression can be used as tools in a body query. Thus, those used in medicine, marketing, sports, or gaming, for example, are susceptible to be repurposed while their original uses serve as know-that context for interpretation. Such devices include, but are not limited to: electroencephalographs, electrocardiographs, electromyographs, thermometers, skin conductance meters, pupilometers, inertial measurement units, and eye trackers.

Devices may also be grouped to provide more complex descriptions of body states. Cruz-Garza et al. [5] present a protocol that combines electroencephalography, inertial measurement units, video recordings and behavioral analysis to study the development of neural networks in freely-behaving human infants. Body area networks [4] are another example of devices grouped to gather information from different bodily sources and produce a coherent description of body state.

3.3 Documentation

The goal of a body query is to produce documentation about the process, that is, a set of interpretations of signals obtained at a given moment during practice. As explained before, these interpretations must be made according to criteria contextualized by the practice and by the know-that that supports the query tools. To illustrate, we offer an example cribbed from the dramaturgical practice of the playwright in our team while using his eye-tracker enabled communication system.

The eye-tracking device translates user’s gaze to positions on a screen coordinate system. Changes in the position are in turn translated to navigation and selection commands upon a fixed menu according to predetermined rules; this means that the signal is interpreted according to the characteristics of the device. We add a new piece of interpretation related to practice by combining information about the sequence of commands with the selected characters during the writing process. By doing this, we fulfill the body query and produce documentation that can be used to reflect upon both the practice and the usability of the communicator system. Figure 3 shows a word annotated with the following interpretation of eye movements: the number of dots represents the ratio between the amount of choices made to select the letter and the minimum choices required to select that letter (one dot represents flawless use, more dots represents more choice errors).

Fig. 3.
figure 3

Example documentation stemming from a body query based on eye movements.

Full size image

This illustrates the possibility to reflect upon the practice: an increasing number of dots as the writing progresses may be interpreted as fatigue from the user or a bug in the system. Such interpretations can then guide adjustments before the next practice session while further body queries can be used to analyze what effect those changes have in the practice. This kind of documentation may also be delivered as part of the thick description of the practice, helping build a shared pool of knowledge.

4 BCI in PaR

While a variety of signal types can be used, we are particularly interested in body queries using the brain as bodily source. Wadeson et al. [20] offer an overview of artistic BCI where they identify four types of systems: (i) passive, those that respond to signals but do not require interaction of intention from the user; (ii) selective, those that allow the user to interact but do not allow direct control of the output; (iii) direct, those that enable users to choose specific outputs; and (iv) collaborative, those that allow multiple user interaction through subsystems that fall under the previous types.

Each of these types can be seen as incomplete body queries that must be completed by practice-related interpretations specific to a project. As practice requires constant reflection upon itself, the results of those body queries can be used by a practitioner-researcher as raw material, introspection information, or control commands for artistic tools that affect the next stage in the doing-thinking process. Thus, we have three interlocked loops: practice, body queries, and the BCI system itself (as defined in [21]). Figure 4 shows these loops and how they interact.

Fig. 4.
figure 4

The interlocked loops model of BCI in PaR. Each loop contains an inner loop that is repeated when the outer loop reaches a certain stage. In turn, each inner loop yields information that is used in the following stages of the outer loop.

Full size image

The PRACTICE outer loop begins with the preparation of the activity: contextualization from literature-practice review, reflections from previous iterations, planning of the practice session, etc. Afterwards, the practitioner-researcher undertakes the creative activity and, while performing, one or more concurrent or sequential body queries can be executed. Each BODY QUERY is an inner loop started during the activity and gathers information from a bodily source through the use of a tool (in this case BCI) while adding automatic interpretations, if possible, to produce live documentation that may be used as real-time feedback. After the activity ends, further interpretations can be assigned to the results of body queries and the resulting documentation is used in the critical reflection stage. All the loops repeat as needed.

It is important to note that body queries are central to this model and that their definition is flexible enough to allow for using BCI either as an integral part of the practice or as a process documentation tool. In the first case, BCI is used as a tool for artistic practice during a research inquiry in a PaR process. In the second case, the use of BCI is akin to the use of video recordings, journal annotations, etc. One case does not exclude the other, however. If used as part of the practice, BCI can also be used as documentation tool; if used as documentation tool, the dynamic nature of the process may suggest using it as part of the practice.

4.1 Body Queries Using BCI

In PaR processes that use BCI, body queries should be designed around existing tool-related interpretation paradigms. By this we mean that the practitioner-researcher must review previous work on BCI to identify the kind of information gathered by different techniques and how it can be mapped to the practice and documentation of process. To accomplish that, it may be useful to frame the design of body queries according to the set of possible commands afforded by the query tool [2]. In this way the practitioner-researcher may codify previous work in a format that enables them to reflect upon the possibilities open for practice while decoupling each particular body query from the technical implementation details of its query tool.

4.2 Documentation Possibilities

As stated before, documentation resulting from body queries may be used in real-time during the creative activity, afterwards during the critical reflection stage, or as descriptions to be shared with the community. Which kind of use the practitioner-researcher makes of it depends on the aim of the practice, implementation feasibility, and interpretation possibilities. The first item refers to the fact that documentation should contribute to the thick description of the practice in the context of the research inquiry. The second, implementation feasibility, deals with more technical aspects of the body query and whether they can be ethically, safely and timely implemented for use during the creative activity. Finally, the possibilities of interpretation refer to which interpretations may be automatically assigned to information coming from the query tool, either in real-time or during a post-process step; which interpretations should be assigned manually by the practitioner-researcher; and the methods to do so.

Beyond the possible ways in which documentation may be realized, it is useful also to think about it in terms of its purpose. Given a set of body queries, each with its own set of interpretations, those interpretations may be codified in order to achieve one of these goals:

 

Transcription: :

Codification occurs sequentially in time to produce transcriptions of the activity. This process may occur in real-time and the resulting transcripts dynamically presented as feedback for the practitioner-researcher. Example of this kind of documentation are Miranda’s Activating Memory [14], a piece for a string quartet and a BCMI quartet, and Cádiz and de la Cuadra’s Kara I [3], a piece for flute, violoncello, BCI, computer music and visuals. In both cases, the signal from the performers is codified as a musical score in real-time.

Annotation: :

The codification is incorporated into the piece created during the activity. In this way the piece is annotated with the results of the body queries and the documentation is embedded on the objects produced by the practice. The documentation described in Subsect. 3.3 and Fig. 3 is an example of this.

Insight Identification: :

Codification is oriented to identify inflection points in the practice; moments when one or more conditions about the practice are met. These conditions are defined during the preparation stage according to known characteristics of the query tool and the practice. The research with jugglers by Schiavone et al. [19] suggest a way to differentiate between expert and non-expert jugglers according to the spectral power in certain frequency bands. In this case, a practitioner-researcher might be notified when the spectral power profile changed from non-expert to expert. This example will be further explored in Sect. 5.

Multiple Performer Relationships: :

If there is more than one person involved in the activity, codifications may be extended to include information about participants (either practitioners or audience members) with the goal to describe their interactions as the practice progresses. Any of the previous examples could be extended to include information about each performer.

5 Examples

To illustrate aspects of the framework described in the previous sections, we will review previous uses of BCI in research projects involving artistic practice. For each one, we will show how some aspects of the model are exemplified by each project and suggest possible documentations approaches based on the available information.

5.1 Eaton et al. The Space Between Us

This example is based on the report of a live performance of the musical piece The Space Between Us used in the research of affective states of performers and audience members [6]. The main goal of the BCMI system described was to “move the affective states of two users closer together, creating a shared emotional experience through the music that is based on the emotional measures extracted from the EEG”.

To accomplish their goal, the authors used existing literature to create a coordinate system of possible affective states and then defined a mapping between alpha and beta band power from an EEG signal to the coordinate system. Each affective state is mapped to a set of pre-composed musical sequences. During the live performance of the piece, EEG data is obtained for a performer and an audience member, their affective states measured and, using some of those states, trajectories upon the coordinate system towards a target affective state are built. These trajectories are used to select musical sequences that are performed.

To summarize, the authors describe:

  1. (1)

    A device for capturing EEG signals from the brain.

  2. (2)

    A set of electrodes placed across the prefrontal cortex.

  3. (3)

    A mapping from EEG signal to affective state coordinate system.

  4. (4)

    A mapping from affective states to pre-composed musical sequences.

  5. (5)

    A display system for the selected pre-composed musical sequence.

Each item in the previous list can be contextualized as part of a body query: (1) is the query tool, (2) is one bodily source, (3) is the query tool related interpretation criteria, (4) is the practice related interpretation criteria, and (5) may be seen as an instance of real-time documentation fulfilling a transcription goal.

From the information available in the paper, we suggest another type of documentation that may fulfill the transcription and multiple performer relationship goals. As the aim of the research is moving the affective states of the users along trajectories to reach a target state, these trajectories may be visualized upon the affective state coordinate system. Figure 5 shows a possible diagram evidencing affective state dynamics during a live performance.

Fig. 5.
figure 5

An example of trajectories over an affective state coordinate system. The horizontal axis is valence (negative and positive) and the vertical axis is arousal (from low to high), each cell represents a possible affective state resulting from their combination. The gray dot is the initial state, the white dots show the proposed trajectory, and the black dots are the real trajectory. We have purposefully left out dimension labels to emphasize the visual aspect of the documentation.

Full size image

5.2 Zioga et al. Ehneduanna: A Manifesto of Falling

The second example builds upon Ehneduanna: A Manifesto of Falling, a real-time audio-visual and mixed-media performance that uses of multiple BCI systems [22]. According to the authors,

the performance is an artistic research project, which aims to investigate in practice the challenges of the design and implementation of multi-brain BCIs in mixed-media performances (...) and develop accordingly a combination of creative and research solutions.

Although there are more participants, we will focus on those fitted with EEG devices: an actress and two audience members. Raw data is processed to obtain frequency band information for each participant, the frequencies selected are supposed to be meaningful in the context of the performance: beta and lower gamma associated with intense mental activity and tension, alpha to a relaxed and awake state, and theta to both deep relaxation and emotional stress. This information is then mapped to RGB color values: beta and lower gamma to the red value, alpha to the green value, and theta to the blue value.

The work is structured in two parts divided in five scenes total. An actress performs on stage with the exception of scene 4, when she leaves the stage and returns at the beginning of stage 5. Fragments of texts and videos are projected on a screen and the video stream is filtered with a color determined by the signals from the participants: scenes 1, 2 and 3 use information from the actress; scene 4 use information from two audience members; and scene 5 uses information from the actress and one audience member. In scenes where there is information from two participants, each color filter is applied to one half of the projected image. In scene 5, the colors gradually merge towards an average color. The color dynamics during the performance “not only correspond to a unique real-time combination of the three selected brain activity frequencies of multiple participants, but also serve as visualisation of their predominant cognitive states, both independently as well as jointly”.

As we did in the previous example, we can summarize the process:

  1. (1)

    A device for capturing EEG signals from the brain.

  2. (2)

    One electrode placed in the prefrontal lobe.

  3. (3)

    A mapping from EEG signal to frequency band associated with cognitive states.

  4. (4)

    A mapping from frequency bands to RGB values.

  5. (5)

    A live display influenced by RGB values.

Once again, we can view each item as part of a body query: (1) is the query tool, (2) is one bodily source, both (3) and (4) are query tool related interpretation criteria, and (5) may be seen as an instance of real-time documentation. In this case, however, the resulting documentation fulfills an annotation goal. While the previous example used the documentation as a transcription used by the performers, in this case the documentation is incorporated into the performance and serves to manifest both transitions between cognitive states and synchronization/desynchronization between the cognitive states of the participants.

The body query illustrated by this performance may be used to fulfill other goals. For example, in a project where the research inquiry is related to achieving certain cognitive states during practice, the resulting live documentation may fulfill an insight identification goal. In that case, the practitioner-researcher predetermines a target cognitive state (which then has a corresponding color or set of colors) and starts experimenting with the effect of certain actions on the projected color.

This example also shows that query tool related interpretations are not restricted to one discipline and may be built from several know-that pools. On one hand, frequency bands are selected according to known associations to cognitive states, an interpretation coming from neuroscience. On the other hand, frequency band mappings to colors were chosen according to “historically established in the western world cultural associations of specific colours with certain emotions". In this way, two different sets of previous knowledge are joined in a common interpretation for the body query in the context of this particular performance.

5.3 Schiavone et al. Towards Real-Time Visualization of a Juggler’s Brain

The last project we review involves research on a particular artistic practice rather than research through artistic practice. The aim of the authors was to monitor the brain activity of a juggler while performing, with the long term goal of creating real-time visualizations of that activity [19]. Two subjects, one intermediate and one expert juggler, participated in the study. In the first part of the study, their EEG signals were recorded during five different conditions; in the second part of the study, the expert juggler was asked to perform three juggling patterns of increasing difficulty while their signal was recorded. All analysis was performed offline.

Following results from previous research, the authors conducted two types of analysis on the signal: power spectra across frequency bands, and spectral coherence between pairs of electrodes. Their results suggest that these measures may be used to differentiate between expert and non-expert jugglers, and between difficulty levels of different patterns.

This example shows an incomplete body query: it has a bodily source and a query tool but lacks interpretation criteria and documentation. We selected it because it illustrates how a body query may be designed from existing research. In this case, the results point the way towards query tool related interpretations regarding the experience of the performer and the difficulty of the juggling patterns. Further research along this line will provide a more robust foundation upon which the interpretation can be built. At that point, concrete documentation possibilities will appear.

We imagine, like the authors, the possibility of real-time visualization of the identified biomarkers. This kind of visualization may work as documentation to guide practice sessions during a PaR process, to annotate live performances to evidence the brain activity related to juggling, or as a training tool for non-expert jugglers. The body query suggested by this research stands in contrasts with those described in the previous examples: while they used BCI as part of the medium in the practice, this would use BCI as an introspection tool regarding the practice.

6 Discussion

We have presented a framework for using BCI tools in PaR, which is centered on the concept of body queries and how they generate documentation that fills a perceived gap in the types of descriptions available in artistic practice. The model combines PaR concepts with BCI-based body queries to provide information to be used in “thick descriptions” evidencing the research inquiry.

We believe the reviewed examples show the flexibility of the framework and how can it be applied in future PaR projects. Two of the examples use BCI as an integral part of the creative process and practice and illustrate how BCI-based body queries may be used as part of the practice or inquiry. In contrast, the last example illustrates another possible use of BCI in a PaR process, not as medium but as a means of documentation to reflect upon the practice in the context of the inquiry.

The framework is also useful because it moves the focus away from the technical details of implementing the BCI system towards the PaR process. This is important to avoid technoformalism [8, cited by 22], that is, to avoid focusing exclusively in the medium while abandoning the artistic concept that guides the research.

However, as we are focused on brain body queries, we must beware of reductionism, as each body query yields information of part of the body state. Practitioner-researchers must be careful when interpreting brain activity, it may be possible for a practitioner-researcher to build up a set of interpretations for a specific project but those interpretations may not be necessarily transferrable to another project. Also, agreed-upon interpretations based on neuroscience literature must be carefully weighed to avoid reductionism, and, ideally, interpretations should include multiple body queries.

We know future work should be done on body queries in general (expanding body-area networks to PaR, for example), nevertheless, we believe this framework represents a starting point in using BCI as a tool for PaR, possibly opening further lines of inquiry, both in each particular project and in PaR in general.

Finally, we want to emphasize that, the feedback and interlocked loops of doing-thinking, querying the body, and reflecting, illustrate the processes by which the metaphorical cogs of the practice are increasingly exposed as the tacit is made explicit.