Keywords

1 Introduction

In this short paper we discuss an ongoing research project that focuses upon the use of landscape as an artifact, which supports the design of interfaces for musical expression. We take a somewhat experimental and unorthodox approach, bringing together ethnography (sensory [16] and autoethnography [4]), participatory design, and “research through design”. Indeed, a key motivation of the work is to understand the role that designing of such systems can have in terms of ethnographic practice, as we shall later explain.

The research is not only focused in the resulting physical artifacts that emerge as part of the project, but also the processes and participation that enable and support creativity and innovation, feeding into design practice. As designs, theories and ideas come forward that they will not necessarily form any sort of logical iterative design process from an external viewpoint, and this is why it is important to unpack and describe some of the issues that emerge throughout the design process. Key to the success of the project will be our ability to offer a way for people to express themselves musically, in a way that enables them to define, understand and control.

As Pinch and Trocco [15] write, “…it is not possible to design a musical instrument by beginning with an objective set of performance specifications. Rather, a musical Instrument design usually begins with a designer’s intuition… Where does that intuition come from?”

Our focus on the landscape as a catalyst for developing an interface from musical expression has emerged from a series of our earlier publications across domains and was influenced by Ingold’s work [9] on landscape. We wanted to imagine expressive systems that could engage a performer directly with the landscape whilst performing. As Ingold suggests, landscapes are ‘practice-scapes’ rather than ‘vistas’ or flat visual depictions. This, he argues, is “…an alternative mode of understanding landscapes, based on the premise of our engagement with the world, rather than our detachment from it” [9]. We shall come back to this notion of ‘engagement’ later. In many ways engaging with our landscape leads us to reflect upon ourselves in and as part of the ongoing construction that is, us being part of the landscape. “…this life process is also the process of formation of the landscapes in which people have lived” [9]. Indeed, in etymological terms, the suffix “–scape” is derived from the Old Norse ‘scyppan’ – to create, to shape. Tracing the etymological root of the term also suggests our interpretation of what a soundscape can be: a creative engagement, a shaping, an engagement with the world, or a trace of other people’s engagement with a their culturally embedded worlds of sound. Importantly, both –scapes do something to each other: being in the landscape does something to the soundscape and the soundscape does something to the landscape. Using the landscape is a way of shaping musical expression and enabling people to use their experience and understanding as a way to mediate and perform.

One of our aims is to draw on the way that we engage with the landscape, mapping these into forms that enable and create mechanisms and opportunities for expressive musical interaction. This may sound like a radical proposition, but in many respects we are surfacing the mundane [17] aspects of our lives in different ways. This is expressed through landscape-based musical artifacts that relate back to lived experience, which like any other ‘documented’ expression of experience is an abstraction. Our approach understands this, making evident the understanding that the expressive nature of the instrument is musical, itself in turn an expression, an abstract manifestation. However, this is not to say that it is not possible to invoke a shared experience or feeling through music, and in turn we would hope to fold such expressions into a more sensory-ethnography [16] inspired understanding of musical instrument/interface design. Later in the paper we provide some conceptual designs that form part of our initial ideation phase.

1.1 Ethnography: A Note

In order to appreciate the methods and tools that we have used to study this we first offer a brief insight into the world of ethnography (within the confines of this paper), before moving into the context of the site and the design of the system.

In this particular instance we focus upon the use of design ethnography, that is when ethnographic methods are used as part of the design process, with a special focus on the experiences of people in relation to the location. In order to explicate our approach, we use Goffman [7] as a reference point who, when discussing fieldwork in his book Asylums, states that, “any group of persons-prisoners, primitives, pilots or patients - develop a life of their own that becomes meaningful, reasonable and normal once you get close to it … a good way to learn about any of these worlds is to submit oneself in the company of the members to the daily round of petty contingencies to which they are subject.” With this in mind we must turn to ethnographies of the site specific, self and the sensory as others [16, 18] have. “Sounds… are not merely abstract acoustic events” [19] their interpretation and use is social, and expressive.

As LaBelle [10] quite aptly argues “sound is intrinsically and unignorably relational: it emanates, propagates, communicates, vibrates and agitates; it leaves a body and enters others; it binds and unhinges, harmonizes and traumatizes; it sends the body moving, the mind dreaming, the air oscillating. It seemingly eludes definition, while having profound effect”. This move to the expressive ‘instrumental’ nature of landscapes to support ethnographic practice is where our interdisciplinary research resides, as a way to understand the more humanistic influences on musical instrument design (intuition) [5].

2 The Site of Expression

Choosing a site and community to work with is difficult, and carrying out research “in the wild” is resource intensive, with the real world often impinging upon the research. Our approach, based on previous research projects is to use an embedded researcher, who is part of the community, understands the site and has experience of using and designing audio performance-based systems. The community that we are currently working with is based in West Wales (UK) and has a hill fort in the area that they live. More specifically there is a history project based around the hill fort and we have been able to attend sessions and take part in the activities relating to the site.

There are also non-governmental organisations that maintain the site and other organisations with an interest in it: as a tourist location, a historical site, and a managed wildlife site. There are data available relating to the site that might be used as a mechanism to impact upon the interface. This multiplicity of stakeholders and data provides a rich set of resources that can be pulled into the design. It is key that we understand some of the issues and data that are created and used by such groups as this data may be called on as a performative tool by which to express one’s self.

The images (Fig. 1) show an aerial photograph of the site and an image of a 3D model created from LiDAR data, which is openly available. Having access to the LiDAR data means that models can be quickly made, printed as physical 3D models, or virtual models imported for example into Max/MSP. The LiDAR data can also be used to alter the pitch, volume, and chosen sound and could also be used as a triggering mechanism. At this stage of the project we are using the shapes contained within the site as a way through which we can start the initial ideation or design phases of the project, enabling us to ask people to engage in the design process and develop early prototypes.

Fig. 1.
figure 1

The two images above show an aerial photograph (left) and a virtual 3D model created with LiDAR data.

Full size image

It can be difficult for community groups to engage with concepts such as sound control, gesture and in this case the landscape as something that can form an interface for physical expression. Our work enables people to understand and engage in concepts, while we acknowledge that we may be guiding the design by providing a framework and some sort of motivation for their explorations. This makes it vital that an experienced researcher is present. In our previous studies taking a probe, or provotype [1] has been key to enabling users to engage with the technology in order for researchers and designers to understand both the actual and potential use of the instruments and software being designed [11] and the role that such groups can play in supporting and developing tools to support existing communities of practice [12]. In many respects this move from the societal, to the institutional and to the self is mirrored in the way that Human-Computer Interaction and Computer Supported Cooperative Work has moved from the lab, to understanding industrial organizationally embedded systems, out into the real world and ultimately into the realm of the maker and a world where one can design and develop systems for oneself.

A recent workshop [2] found that using tangible artifacts as a way to both use and interact with non-tangible media meant that we were able to collect sounds, to explore their nature in respect to their interaction with other people and machines, and to study their integration with a physical format. These findings led us towards issues relating to physicality, performance and the expressive nature of audio and landscape in a physical format. We used Percussa AudioCubesFootnote 1 as a way to enable people to bring sounds together easily and to explore contrast and similarity of sounds, pleasurable and unpleasant sounds, affectively engaging and uninteresting sounds, sounds that evoked memories, sounds that were unrecognizable, and many more. We see this as a move to the expressive ‘instrumental’ possibilities of landscape to support ethnographic practice in regard to developing an understanding of such sensory resources for expressive systems design. The purpose of fusing ethnography and musical expression is to explore the intersection of the arts, sensory ethnography [16], phenomenological aspects of place experience (Genius Loci, the spirit of place), and the discipline of human-computer interaction as a means of conceptualizing, building, and evaluating a new musical interface. What might be gained from rendering knowledge about culture in a more generative form, through the collaborative and creative process of ‘playing a landscape’ as an instrument?

From our perspective our position is further expounded by the performative nature of Goffman’s ethnographic approaches, that in many ways leads us to reflect upon ourselves in and as part of the ongoing construction that is us ‘doing’ being (to borrow from ethnographic terminology) part of the landscape. Taking part in, being and expressing ourselves in a dramaturgical way.

The discipline of anthropology also maintains a long-standing engagement with sound and how acoustic cultures or worlds are performed [6]. Sound matters in the composition of both cultures and bodies. We do not propose that sound should be sequestered from an overall multi-sensory perspective on what it means to experience. Yet emphasizing our concern with sounds and acoustic environments can be a useful counterpoint to the overwhelmingly visual-centric epistemologies of western science.

3 Sonification

Sonification, the “transformation of data relations into perceived relations in an acoustic signal” [29] provides an alternative way of exploring the landscape but raises challenges for design. Where soundscapes, such as Virtual St Paul’s CathedralFootnote 2, simulate a scenario using data, sonification may be used to display more abstract concepts, such as the environment, archaeology and the land.

The technique is a way of interacting landscape through seismology [25] but might be a useful way of interacting with archaeological context. Using the soil contexts, we can build a time based sonification demonstrating how the layers have developed and whether there are anomalies.

Environmental data that may be sonified includes the weather or soil conditions. The complexity of these variables poses issues with the complexity and volume of the information as this will include humidity, temperature and wind speed amongst others [30].

Networks, such as power and telephone lines, traverse the landscape; some lines visible, others not. Home appliance electrical has been sonified [26, 27] but not at the national level. The national academic network [23] is sonified in various methods to show how part of academia’s fabric using white nose and identifying the sections. The sonification might show how modern and ancient landscapes co-exist in time and space as layers, waiting to either be discovered or made invisible. This might also be extended to maps of land use and boundaries to show the changes over time.

In previous work, reported performances of David Garrick, the Eighteenth-Century actor [22], were sonified. This allows us to build a model to test how assumptions about a voice might act within a space, such as a theatre where the building specifications can be represented. Using a hill fort poses challenges through the way that a voice or set of voices might work within not only the physical factors but also the environmental ones. As the fortifications are open air, weather and direction changes how sounds may be heard or interpreted.

This raises the question of how to sonify the potential movements of a particular person or people and the audio calculations between the listener and the sonified object. It would suggest the sounds should reflect not only the actual space but also the historical aspects of the location.

A consideration is whether the sonification has interactive experience or not and how this may be provided. Two choices are immediately apparent: mobile phones or wearable devices. Using an application, mobiles could work to provide a sonification unique to each phone and may work in concert with others [24]. Wearable devices, such as the MozziDuinoFootnote 3 could be used to allow humans to interact the environment without relying on being able to download and application. In both cases, the machinery might be programmed to understand the landscape and signals. There is also the question of leakage - is each listener hearing something for someone else and does it intrude? – or creating a social instrument.

This provokes questions about how the sonification is controlled. Work with museum visitors with visual impairments [23] concluded that non-linear, interaction with similarities to phone interaction helped discover how the system worked. It also allowed the testers to navigate the audio in their own manner, discovering the narrative that is relevant to them as well as navigating between introduction and description. It suggests that the ethnography and participatory design are useful tools to develop interaction models but that a non-linear interaction may be more meaningful. They might be more individually meaningful if interacting with both human and landscapes.

Limitations on the interactions limit the facets and, potentially the complexity, that can be shown. This also imposes the sonification onto the user, directing the attention. Conversely, too many would be too difficult to understand.

These various aspects present challenges for design in terms of the number, and complexity of each, aspect. This has a symbiotic relationship with designing the interaction and the number of layers that we may consider, both from a technical and a human perspective. This may create technical issues with designing the types of interaction, such as buttons or beacons, and the person understanding the amount of audio information as well as potentially visual information. Consideration of these factors will affect the sound design.

4 Ideation, Data and Design

Shifting towards an understanding of the soundscape as an engaged encounter (a shaping, a practice rather than a depiction or a –scope) opens up a number of interesting possibilities. The main challenge is to set the scene for a participatory and an engaged process of ideation with the relevant stakeholders. A secondary challenge is to start to think about and envision how systems such as this might inform design and the development of tools that can support the development of cyber-physical systems.

First, the intention is not to create a static soundscape. Rather than considering the soundscape as an artifact or a ‘depiction’ of an existing structure, how can land- and soundscapes become part of a musically interesting expression? How do we create the possibility of interacting with the landscape as an instrument?

Some of the indicative questions to explore include:

  • How can we collaboratively explore the landscape with the purpose of finding out what is significant and relevant to the performer?

  • What features of the landscape are interesting and meaningful to manipulate for an audience?

  • What are the potential affective connections an engaged land/sound-scape and how can they be augmented or made interactive in interesting ways?

  • Physical (direct embodied manipulation or physical movement) or ‘ephemeral’ (abstract data, “invisible” sensor data,) interfaces, what is the role of the body in interaction?

  • Distance of interaction (what is a meaningful “shape” of an interactive landscape, when does it become meaningless or un-engaging for performer and audience?

  • Discrete experience or event. Or both?

  • Self-design as way of cultivating relations to other (and vice versa?)

  • A musical instrument as an artifact of relations vs. an artifact of meditation or personal growth?

  • How do we sense time/progression of a (temporal long-form) musical piece, how does the time/rhythm of a piece relate to local times/rhythms?

  • What are sources in the landscape of fascination, mystery, hauntings, longing, loss, love, life, death, decay, activity, rest, congregation, dispersal and so on?

  • What does history do to a landscape – local vs. non-local vs. global narratives about place.

  • The experiencing body or the topographic abstraction as parameter in interaction.

4.1 Interfacing with the Past as Resource for Expression in the Present

Landscapes, the land that is formed by human activity, are rich with historical markers. These range from the incidental (an old path) to the monumental (a monument or any other built artifact that aims to prompt a distinct historical awareness). The land is haunted by the ghosts of personal and wider cultural memories [4]. Both personal and cultural histories disappear or are buried under layers of other histories. Some are distorted (glorified, demonized or otherwise mangled) by narratives that somehow displace their significance or dispute their truthfulness.

In an experiment under the banner of “Numbers into Notes” [3], we have installed musical devices within an interactive space. Each device generates music according to algorithms, which are parameterized by sensor readings and physical interactions within the space. This challenges the traditional boundaries of composer, performer and audience; rather, each interaction is an intervention, which influences the experience of the next audience member. It regards the landscape as an interface to the co-created composition and experience of music.

Furthermore, a historical dimension to our work bases algorithms on the mathematics of the 1800s. Essentially we are conducting an experiment, using the methodology of “experimental humanities”, in which we explore the idea that mechanical music boxes from the time of Ada Lovelace, have evolved into mutually responsive digital devices today. It is the making manifest and the bringing into being that has thus far inspired compositions, performances (opera) and the design of (physical) digital music objects. It is this move from the intangible to the tangible, and the possibilities that this affords in terms of expression and experience that bear upon our design understanding.

The image above (Fig. 2) starts to map out, and physically illustrate, how the forms that exist in the landscape and archival imagery relating to the place can be pulled together and imagined as catalyst for conversation and ideation. Using tools such as these, which bring together archives, ideas, a range of technique and personal reflections, all from disparate sources both allow people to imagine what can be and envisage what is possible. In the next section we discuss some of the technical possibilities: although not the key motivation of this research, it is important as a feature of our endeavors. It also illustrates the multitude of technical possibilities that are available to contemporary developers and designers.

Fig. 2.
figure 2

Features of the hill-fort outlined as control zones, historical images added to support ideation.

Full size image

4.2 Future Forms and Plans

As we have seen, we are currently engaging with a virtual 3D model of the site: we have images, maps, drawings and recordings. In our next phase of development we will further explore the landscape with people in order to deep map [Pearson Arch] the space. Our research through design [20] builds upon existing work, by taking a fully multidisciplinary participatory approach. Understanding the many layers of meaning when designing such systems plays an important part in the player being able to express themselves and the audience being able to understand the nature of the expression. Our discussions based in what is technically possible have ranged far and wide, opening up a range of opportunities and possible challenges, when related back to the landscape.

For example using sensors that might feed back to the interface, perhaps based on the weather, as Ingold [8] maintains, the weather connects the sky and the earth, and is a significant part of the identity of a landscape and the people who live in it. However being able to manage an ever-changing interface could be difficult, equally an interface that was based on such weather data could equally prove less than expressive, it the weather was calm. How might we design for change, for transience, for things that sometimes are a key part of the landscape and at other times disappear? In terms of building physical interfaces this is a problem, but easier to accomplish in software-based systems. Equally thermal interfaces could be used to convey the seasonality, which could mean that a physical interface could be hot, or cold to the touch, having a dramatic impact upon the way that player/s would perform. Hardware and software could be combined: as we have already noted, it is possible to pull in the virtual 3D model into Max/MSP and experiment with implementing designs for musical expression; it is also possible to 3D print the landscape in different materials and add textures or images, where the images need to be permanent, but could be mapped over to another controller or existing system in the first place in order to experiment with different layers and the player’s responses to different layers, techniques and sounds. The layers could consist of anything from wildlife data, to personal reflections and even geophysical data that could enhance different ways of thinking about what the performer could express. Our research has also led us to start to look at other platforms such as opera [30] as a mechanism to use and stage interpretive pieces and the use of intelligent agents [31] as a way to support and assist people as part of the process of musical creation

Currently we have a large range of possibilities ranging from using ferrofluids for emergent interfaces, to software systems and physical interfaces and there have even been some discussions about the use of Hertzian space seeFootnote 4. As such even discussing and imagining what it might be possible to realize is creating a rich resource for design.

5 Conclusion

Using the landscape as an artifact that can be used, reflected upon and used to imbue and develop new forms of expressive musical interaction is both interesting and challenging. It is clear that understandings of landscape are multi-faceted and as such we hope that this short paper has started to raise some important issues that relate to this truly multidisciplinary space, that needs to be understood in terms of developing social, physical and technical understandings of what can be accomplished. We discuss the initial stages of the project that brings together understandings of landscape and people in order to further the ways in which expressive interfaces can be designed and developed. Even in these initial stages of the project important issues and possibilities are highlighted that we propose other researchers and designers will find of interest. As such we would hope that this short paper serves as a platform for reflection and discussion.