Keywords

1 Introduction

The Vibropixels are a scalable, wireless tactile display system whose reconfigurable nature allows for use in a wide variety of applications. While the system was initially developed for use in professional artistic productions, it also suitable for use in other applications, including tactile notification systems and virtual reality environments.

This paper consists of two sections. In the first, we describe the Vibropixels tactile display system as it was used in the artwork Haptic Field. We will briefly review tactile display applications and implementations, and review the requirements of the creation of a system designed for use in professional artistic productions.

In the second section of the paper, we discuss the use of the system in the artistic creation process. As the system was designed in a collaborative research projects with professional artists, one of our interests was facilitating the exploration of the system throughout its development stages. One way in which we approached this was allowing for easy access to and modification of the system’s hardware, firmware, and software components. Our intention was to allow transparent access to the system’s functionality as well as avoiding predefined limits on the its use. While our primary concern in this project was the creation of an artistic work, the flexibility provided by these design decisions may also facilitate the system’s use in other applications.

2 The Technical Development of the System

The first public use of the Vibropixels was in the artwork Haptic Field, which premiered at the Chronus Art Center in Shanghai, China from July 9–September 6, 2016. Created by Chris Salter and Tez in collaboration with Ian Hattwick, Haptic Field is an immersive multisensory art installation based upon an earlier work by the collaborators, Ilinx, with visual, sonic, and tactile elements [3].Footnote 1 In Haptic Field, visitors wear a garment with seven Vibropixels attached and navigate a large, dark space while experiencing a variety of tactile patterns.

In this section we review wearable tactile display systems and implementations, the requirements placed upon the design of the Vibropixels by their intended use in professional artistic productions, and a technical description of the system as used in the premiere of Haptic Fields.

2.1 Wearable Tactile Displays

Tactile displays systems have been created for a wide variety of applications, including navigational assistance [2], sensory substitution [1], and enhancing interaction with virtual environments [5]. Many tactile displays are either embedded in handheld devices [7] or in wearables limited to one segment of the body, such as a ring, glove, or belt [12]. Other displays are able provide stimuli over a large portion of the body, frequently taking the form of a jacket or vest [4]. However, it is difficult to create such a system that is both robust enough for use in public presentations and also inexpensive enough to scale easily.

Another challenge is that tactile display applications often have specific requirements for actuator locations. Lindeman presents a vest created to provide feedback to participant’s interacting with a virtual environment [5]. One of their requirements was to provide feedback when an avatar collides with doorframes, necessitating actuators placed on those parts of the body most likely to contribute to such collisions such as the outer shoulders.

The use of tactile displays in wearable form factors suggests the systems be embedded within a garment. Frequently, actuators will be distributed within the garment with wired connections to electronics which provide signals for driving the actuators as well as which allow for communication for a remote computer to control the display [11, Chap. 4, p. 11]. This architecture presents several problems for prolonged use. First, the wiring greatly impacts the garment’s bulk, weight, and flexibility. Secondly, the wiring present potential points of failure as it often will be exposed to stress both when the garment is put on as well as in normal movements. Third, this construction can be time-consuming and expensive. Lastly, this architecture tends to locate the weight and bulk of the battery and wireless communications at a single point.

By switching to a system comprised of a monolithic, self-contained devices many of these problems are eliminated. However, the tradeoff is the increased size of the hardware at each actuator location. Therefore, ideal applications for the Vibropixels will be those which require a coarse distribution over the body rather than high spatial resolution.

2.2 Requirements for Use in Professional Artistic Productions

The requirements which shaped the design of the Vibropixels grew out of our intended application of use in professional artistic productions.

One primary requirement was that the system be able to accommodate large numbers of actuators, allowing for the distribution of tactile stimuli across many simultaneous participants. The cost of developing and manufacturing the devices therefore became a priority. We also had an interest in supporting the social context of this kind of artistic experience. One way of doing so was the integration of LED lighting in order to allows participants to get a sense of the tactile stimuli being experience by other participants. Another was the integration of motion sensing which would allow future applications which transmit tactile information regarding other participant’s movements.

Another primary concern in the systems design was both its robustness and ease of use. Artistic installations which last for weeks or months are often maintained day-to-day by untrained staff. The system should be not only robust enough to endure constant use throughout this timeframe, but also should not require any specialized knowledge to run, maintain, and repair in the case of failures.

Finally, we also had an interest in supporting the system’s use for a wide variety of artistic works. Many applications for tactile displays may go unexplored due to the difficulty of creating such systems. Our intention was to create a system which would allow for easy reconfigurability, supporting a variety of applications both within artistic contexts as well as within the broader research community.

2.3 System Overview

The Vibropixel system consists of four primary elements: the hardware implementation, the embedded firmware, the wireless communication implementation, and software tools for generating control messages. In this section we will discuss the first three elements; in Sect. 3.2 we discuss the creation of software tools and user interfaces for use with the system.

The electronic hardware is built around an ATmega328PB microcontroller running at 8MHz, as shown in Fig. 1. For reasons described in Sect. 2.2, we also integrated four individually-controllable WS2812b RGB LEDs as well as a Freescale Semiconductor MMA8652 3-axis accelerometer. A Nordic Technologies nRF24L01+ 2.4 GHz transceiver provides for wireless communication.

Fig. 1.
figure 1

A system overview showing the hardware configuration of a single Vibropixel.

Full size image

As the creation of vibrotactile stimuli was our primary concern, we examined a variety of tactile actuators in order to identify a solution that would be power-efficient, generate strong vibrations, allow for the creation of a variety of stimuli, and be cost-efficient. Our final design utilizes two different kinds of eccentric-rotating mass (ERM) actuators which possess very different characteristics. The first is a pancake-type actuator that is highly-efficient in terms of power consumption and efficiency, and which creates a reasonable amplitude of vibration. We have found, however that this kind of actuator often responds very slowly, with a rise- and fall-time frequently exceeding 100 ms.

In order to allow for the generation of stimuli which require faster response times, the Vibropixels also utilize a cylindrical-style ERM actuator. These actuators tend to be much less power-efficient, drawing three to four times as much current. However, they also respond much faster and are able to generate a higher peak amplitude of vibration.

2.4 Firmware Overview

The embedded firmware on each Vibropixel consists of software modules for actuator control, LED control, power management, wireless communications (discussed in Sect. 2.5), and system management. A module for use with the accelerometer is under development for future applications. The firmware was programmed in the Arduino IDE, which provides for an easy-to-install and easy-to-use hardware programming interface.

Signals for controlling the actuators are created onboard the Vibropixels using an actuator envelope generator with controls for attack time, sustain time, amplitude, and decay time. A secondary function we refer to as oscillation generation helps with minimizing wireless transmissions as well as providing for the creation of more complex stimuli. The oscillation function allows for retriggering the actuator envelope generator at frequencies ranging from 0.01 to 30 Hz. Oscillation envelope controls allow for shaping the amplitude of the output from the actuator envelope generator, with controls for attack time, decay time, and overall length of oscillation.

As the primary function of the LEDs is to provide a visual reference of the current amplitude of vibration, controls are provided for selecting the LED colour as well as for scaling the brightness of the LEDs relative to the vibration amplitude. Figure 2 shows a sample user interface which provides controls for the actuator envelope generator, the oscillation parameters, and the LED parameters. The interface also provides a simple visual feedback regarding the expected output of the current settings.

Fig. 2.
figure 2

A sample user interface which provides access to the control parameters used in Haptic Field, as well as visual feedback regarding the current settings.

Full size image

2.5 Wireless Communication Implementation

The design of our wireless network grew out of the requirements of creating a highly scalable system with no limit on the number of receivers. To accomplish this, our system consists of Vibropixels which act as passive receivers and transmitters which relay control messages from a central computer. One of our primary concerns was to minimize the number of wireless transmissions as our target applications could require many hundreds of receivers. To accomplish this, the system we created:

  1. 1.

    does not require hand-shaking between client and server devices in order to establish the network.

  2. 2.

    does not utilize packet acknowledgement capabilities which allow for confirmed packet transmission. Due to the fact that there is no guarantee that a particular control message will be received by its intended recipients, our implementation requires that each control message be entirely self-contained. All device settings and parameters are either set in firmware, permanently stored in non-volatile memory, or able to be contained within a single control message.

  3. 3.

    utilizes a two-stage device addressing scheme consisting of a group ID and a device ID. A control message can be addressed either a single group ID, or can also be broadcasted to all groups. Within each group, a message can be addressed to any combination of device IDs. This is accomplished by utilizing a device ID in which each bit represents a single device. If that bit is set high then the device it represents is being addressed; if the bit is set low, the device is not addressed. This scheme, as shown in Fig. 3, allows a single control message to be addressed to several devices.

  4. 4.

    allows a single message to create multiple vibration envelopes (as described in Sect. 2.4) with a total duration up to 65 s.

Fig. 3.
figure 3

A simplified depiction of the Vibropixel addressing scheme. In this example, the 6-bit address 101011 indicates the message is addressed to devices 0, 2, and 3 in group 2.

Full size image

3 Artistic Creation with the Vibropixels

Many considerations affect the ability of a hardware system to support the artistic creation process. In this section, we will discuss these considerations, the artistic creation process in Haptic Field, and our ongoing work to facilitate the use of the Vibropixels in other artistic contexts.

3.1 Artistic Creation with Digital Hardware Systems

One of the primary challenges of designing hardware systems for artistic applications is the definition of design requirements. In their overview of software engineering practices in the creation of interactive art, Trifonova et. al found that frequently “requirements are difficult to capture, vague at the beginning and frequently changeable” [10]. This is largely due to the exploratory nature of much of the artistic creation process, in which inspiration comes as much from the unanticipated consequences of a system’s behaviour as from the intended results. One consequence of this, as Trifonova et. al found, is that artists are often interested in expanding their knowledge of software engineering in order to increase their ability to experiment with digital systems.

Considering that many artists will not possess the technical knowledge to program a system directly, Colin Machin argues that one way of facilitating artistic exploration is the creation of simple application-specific programming languages [6]. These can serve as a kind of middle-layer, providing the artist with the ability to change the system’s parameters while shielding them from the underlying complexity. While Machin described these languages as potentially being structured like the BASIC programming language, in our experience developing application-specific tools in common new media programming languages such as Max/MSP or Processing can serve the same function of shielding the user from the system’s complexity while also allowing them to control the system’s parameters algorithmically.

However, the creation of these tools is a research area in itself. Ben Shneiderman argues that the proper design of software tools can facilitate the creative process, and proposed a set of design principles to assist in this, which include supporting exploratory search, providing rich history-keeping, and designing with low thresholds, high ceilings, and wide walls [9]. While Shneiderman’s work is focused on the creation of software systems, these principles also suggest ways of structuring interactions with hardware tools. Supporting exploratory search, for example, might suggest the creation of multiple representations of system parameters, encouraging different perspectives on working with it. Providing rich history-keeping is particularly important for exploratory processes, giving the ability to locate and recall the results of earlier explorations.

Shneiderman’s principle of designing with low thresholds, high ceiling, and wide walls corresponds most closely with the arguments of Machin and Trifonova et al. This principle suggests that a system should provide for interaction and control of the system which is commensurate with the knowledge and intention of the user. In order to accommodate a wide range of users, it is common to attempt to provide an entry point for users with little technical skill, but also allow for access to lower-level parameters and functionality when required. Wide walls refers to a system providing a range of functionality for solving application-specific challenges, minimizing the need for the use of additional systems to fulfill application requirements.

One example of the support of these principles in the music domain is libmapper, a software tool for creating relationships between sensor data from hardware control interfaces and music synthesis parameters [8]. A complete set of these relationships is often referred to as a mapping. In this software, available input and output parameters automatically populate the left and right sides of a GUI, respectively, and users are able to create arbitrary connections between them using click-and-drag techniques. While creating effective mappings is a challenging task, the software GUI allows for quickly and easily iterating through possible combinations of input and output parameters. In addition, once a connection has been made, an additional set of advanced parameters allow for mathematical and algorithmic transformations of the data between the input and output devices.

In the next sections, we will consider the ways in which the concepts described above manifest in how artists are able to interact with the Vibropixels.

3.2 The Creation of Haptic Field

As our primary focus is not user interaction design but rather hardware system development, and our collaborators are fluent with new media software creation tools, we chose to focus on making the system easy to modify rather than attempting to create a fixed user interface. As described in Sect. 2, the system utilized in the creation of Haptic Field is comprised of four elements: the hardware architecture, the firmware, the communication protocol, and software tools for generating control messages. Many of the design decisions for each of these components were made explicitly to facilitate modification and programming of the system by our collaborators: the user interface was created in the popular multimedia programming language Max/MSP; the communications protocol was documented and communicated to our collaborators so that they could program alternative user interfaces for generating control messages; and the firmware was programmed in the Arduino IDE as many artists are familiar with working in that environment.

Following the initial public presentation of Haptic Field, we interviewed our artistic collaborators in order to ascertain how they interacted with the system’s components and the software tools we provided. In particular, we were interested in when components were used as-is, and when they were used as the basis for modifications. The components and tools we provided them and their use is as follows:

  1. 1.

    Hardware implementation: Various iterations of the Vibropixels throughout the prototyping and development stages.

    Use: The Vibropixel hardware was not presented to our collaborators as a development platform, but as a fixed hardware configuration, and in general was used as such in the creation of the work. However, for the final artwork 12 Vibropixels were modified to create permanently installed ambient lighting. For this purpose, they were modified to accept power from a DC power supply, and hung from the ceiling of the installation space.

  2. 2.

    Firmware: Various iterations of the Vibropixel firmware developed in the Arduino IDE were provided.

    Use: While our collaborators did look at the firmware provided to them, they did not modify it or use it to increase their understanding of the system.

  3. 3.

    Written Documentation: Written documentation of the wireless protocol and the control message structure.

    Use: The documentation was used to help conceptualize the structure and capabilities of the system. While the description of the control messages did aid in the creation of control messages without using the provided GUI, our collaborators added that conversations with the system’s developers were perhaps more helpful in that task. It is clear, however, that the majority of the control messages were created with the software GUI as described below.

  4. 4.

    Software Tool: A software abstraction programmed in Max/MSP which accepted control messages and formatted them correctly for wireless transmission.

    Use: Used as intended, and not modified.

  5. 5.

    Software Tool: A simple software GUI programmed in Max/MSP for changing control parameters for the Vibropixels. The GUI provided for the development of Haptic Field was similar to that shown in Fig. 2, but without any visual representation of the envelopes and oscillation.

    Use: Used as the primary interface for exploring the functionality of the system as well as for creating control messages in the final composition. Various modifications were made during the creation process, including: adding support enabling a MIDI keyboard to control device parameters and send control messages to individual Vibropixels; adding algorithmic processes to transform device parameters over time; and adding functionality for receiving control messages from external software controlling the progression of the composition.

  6. 6.

    Software Tool: A software GUI programmed in Max/MSP for generating device addresses in the format described in Sect. 2.5.

    Use: Used as-is to generate device addresses for both the creation of the work and in the final composition. Modifications were made to allow for algorithmic generation of device addresses, and for receiving control messages from external software controlling the progression of the composition.

From these responses we can see that our collaborators primarily interacted with the highest level software tools provided to them. While the documentation of the control message structure did aid in their understanding of the system and the creation of some control messages, time pressure led them to revert to using the provided GUI and directly asking for help. As one of our collaborators reported, “we did look at the documentation but I think at that point we were just trying to get things going and just wanted something as intuitive as possible.” Our other collaborated concurred, saying “when there is so little time ... you want to just see things working and you don’t want to spend so much time on the software.” However, he also mentioned that, given the right context, the ability to work with different layers would be appealing, saying “when you have more time, you’re relaxed, and you want to think of something new then I would definitely go and try .. to modify the software and the source code.”

While part of our strategy for creating wide walls and high ceilings remains providing direct access to each of the system’s components, it is clear that many factors will affect the ability of users to take advantage of that access. Time pressure is one main concern, driving users to use those tools which provide the most immediate access to control of the system, however limited. In addition, users will often utilize those tools which most closely match with their existing knowledge and preferences. However, the ability of a simple modification to the hardware to remove the need for a separate system to provide ambient lighting suggests that access to the individual components remains valuable, but the ability to take advantage of this access is contingent upon time, knowledge, and the complexity of any necessary modifications.

The desire for lower thresholds for getting started with the system was common. One of our collaborators indicated a desire for help documentation such as that provided by Max/MSP, providing small single-function abstractions which would enable him to “copy a little piece and then you change it ... that is the way I’m used to learn things ... [I would like it] if there is a way to make it simpler to approach at first.” However, shortly after he added that there is also a need for an interface which allows access to all of the system’s parameters, saying “but of course you end up using the big thing because that’s what makes it juicy.”

Understanding the functioning of the system conceptually presented a challenge. Being presented with a UI which exposes all of the parameters of the system made it so that “you have to really understand the bigger mechanism.” While one collaborator said “I remember looking at the documentation very carefully in the beginning... in order to understand how that worked and it was very clear and very important to understand,” he continued by saying that it was less useful in the creation process – “and then after it was more like a practical thing, and we were also [working together in the same room]... so I kind of forgot about the documentation, I was looking at the patch and how it worked and just trying things out in the patch.”

Our collaborators suggested that one way of helping resolve this challenge would be to provide visualizations of system behaviour, saying as that “it would be useful to have some way of understanding the correlations between the different envelope parameters”, a problem we addressed in the creation of the GUI shown in Fig. 2, and also asking for a “visualization of groups ... a top-down representation“ of the current state of the system.

Comments such as these made it clear that the lack of visual representations of the system’s behaviour as well as the layout of the software tools negatively affected the user’s exploration of the system – “the tools at the software level could have been more intuitive in framing the paradigm ... some of the stuff could have remained under the hood ... and the more generalizable parameters, amplitude, colour, intensity of the RGB channels, the oscillation features, those things could have been marked out more clearly to use them more creatively.”

3.3 Facilitating Use of the System in Other Artistic Works

One of our goals was for the system to be suitable for use in a variety of artistic and research applications, and several such projects have been on-going since the initial public presentation of Haptic Field. Development of the various components of the system is ongoing, and our goal is to build upon our prior experience to enhance the system’s ability to support these projects. In this section we will discuss several development strategies and the ways in which they engage with Shneiderman’s principles.

The creation of a new software GUI to lower the barrier to entry is one ongoing development. Figure 2 shows the current implementation, which takes into account our collaborators’ requests for visual feedback of the interactions between actuator envelope and oscillation parameters, and also groups the controls in a way which makes their relationships clearer. While we remain focused on hardware development, we are also working with our collaborators on the creation of visualizations to provide feedback regarding the spatial distribution of multiple Vibropixels, and to provide a real-time visualization of vibration amplitude for the whole system.

Several aspects of the Vibropixel hardware are in development, including independent control of the two motors as well as the use of the accelerometer. While we are striving to use conceptually simple implementations for these aspects, their interaction with other aspects of the system can lead to complex behaviours. To assist in their use, we are planning on creating presets fixed in firmware which provide immediately useable settings. In addition, we are going to provide software tools providing access to each aspect’s parameters as well as providing visual feedback regarding the current state of the system.

As we saw in Haptic Field, simple hardware modifications can allow for implementing functionality outside of the Vibropixel’s intended use, preventing users from needing to integrate additional systems into their application. To assist in this, we are planning on a hardware revision which provides access to unused microcontroller pins, providing additional analog inputs for reading external sensors and allowing for communicating with other digital modules using protocols such as I2C and UART. While most applications using these pins will need firmware revisions, we also plan on building into the standard Vibropixel firmware support for basic functionality, such as using analog sensor data to control actuator vibration.

4 Conclusion

In this paper we have presented a description of the Vibropixel system as used in the artwork Haptic Field, and a discussion of the ways in which the system supports the artistic creation process. While our focus has been on artistic applications, much of the discussion is equally applicable to other research applications which could utilize a tactile display system. In particular, Shneiderman’s principles which we have focused on arose out of work on creativity and innovation across disciplines including engineering, research, and media [9]. The observations we have made regarding our collaborator’s use of our system, therefore, should be of interest to the designers of tactile display systems in other collaborative contexts.