1 Introduction

Art and technology are consistently related to each other in many aspects throughout history. Both share a core creative process. With the advent of technology for creative activities, the process of creative expression has been propelled to new heights. One of the newest additions to the suite of creative tools and techniques is vibrotactile art. Vibrotactile technologies enable vibration patterns to be composed so that audiences can experience/feel those patterns through the tactile sense rather than through audio/visual media. Currently there are a number of vibrotactile output devices (i.e. Emoti-chair [1]) but there are few methods for inputting the data to express those vibrations (e.g., [2–4]) and other methods for other media (e.g., music notation and music composition tools) are used. Artists have recently become interested in creating vibrotactile compositions as a new art form. However, in order to do this, new tools are needed so that the unique elements of vibrotactile stimuli can be expressed. In addition, there is no standard notation system for vibrotactile stimuli in which these expressions can be recorded and/or shared with audiences and other artists. This paper presents a vibrotactile time-based notation system and a software tool called Beadbox, which embodies this notation system. Artists use the vibrotactile notation system though the Beadbox software to compose vibrotactile expressions. In doing so, other artists can reproduce those vibrotactile compositions, and they can be represented on a variety of output devices. The Beadbox is evaluated with user studies where participants are asked to use the software to create a vibrotactile composition. For each user study session, pre and post questionnaire are used to gather subjective data regarding their impressions and opinions. Qualitative and quantitative data are analyzed to determine system elements of the usability of Beadbox and its user interface.

2 Background

2.1 Human Vibrotactile Perception

The basic mechanics of the human vibrotactile system are made possible by mechanoreceptor cells in the skin, which convert the frequency, amplitude, and duration of vibration applied to the skin into an electrical signal that travels to the person’s brain [5]. Many researchers have claimed that there are four essential parameters of a vibrotactile stimulation: frequency, intensity (amplitude), spatial information, and temporal information [3]. A vibrotactile notation system then must be able to provide the information relating to these four parameters. The human sense of touch is limited to discerning frequencies between 20 Hz and 1000 Hz (the human ear can perceive frequencies between 20 Hz and 20,000 Hz) [6]. However, several studies warned about the negative effects of low frequency vibration on the human body so care must be taken when applying low frequencies to the body [7]. For the temporal coding, Cohen et al. [8] conducted experiments to find out how the duration of vibrotactile stimuli influences frequency discriminability. The results indicated that the minimum duration of vibration to determine tactile presence is 50 ms. For the intensity of vibration, Verrillo [9] interpreted that the derived curves of vibrotactile intensity change similarly to the equal loudness contours in audition. Displaying these maximum and minimum vibrotactile values where empirically derived and assisting artists in understanding how to work within them is one goal of the compositional tool developed for this research.

2.2 Existing Vibrotactile Authoring Tools

Several authoring tools that allow for vibrotactile compositions exist in the literature. The posVibEditor [10] allows prototyping for vibrotactile pattern by using a timeline with graphical representation of waveforms. Users can edit the frequency patterns of the stimulus in detail. Also, the multichannel timeline interface uses sample clips of frequency patterns with different durations created in the pattern editor. The VibScoreEditor [11] uses a western music notation score as the user interface where each musical note has a single frequency associated with it. However, it can be complex to use if the user does not have background knowledge of music composition structure and format. The minimum duration of a note is about 1 ms, which is hard for a human to perceive. The frequency range does not have a maximum limit meaning that composers could use frequencies that are not detectable by the skin. In addition, the developers of the VibScoreEditor did not conduct any user studies so it is difficult to determine how usable and useful the tool is. The TactiPEd [12] provides a visualization of spatial information as well as frequency, duration and intensity in a histogram format via a timeline. The interface provides a visualization of output orientation, which can be rearranged by users to fit their output device. Each sequence has its own visualization for its output, which is coloured in a different hue. However, there is no visualization for the intensity level and the timeline measure is difficult to match with the actual frequency samples. These existing authoring tools take different approaches to their interface designs, each with advantages and disadvantages. The vibrotactile composition tool developed in this research will consider the advantages provided by these tools and attempt to avoid their pitfalls.

3 System Overview of the Beadbox

The purpose of the Beadbox is to facilitate the creation of vibrotactile interactive art by controlling four essential variables: (1) frequency, (2) intensity, (3) spatial distribution of the signal among the vibrotactile actuators and (4) temporal information. All of these components are the foundation of a vibrotactile signal and can be controlled by a user to produce patterns over time and space that can be felt by human skin [13]. The Beadbox proposes a unique notation system in order to allow users to control these variables and produce these patterns. Users can record a vibrotactile composition, play the piece while they are creating, edit the file, and save the finished piece. Figure 1 displays an overview of the Beadbox with each component labeled. The Beadbox was implemented in Java 1.8 and uses an Audio Stream Input/Output (ASIO) API with the ASIO4ALL driver and jASIOhost library. The output device for the user study is the Emoti-Chair [1] with a Firepod™ soundcard connection. Each Bead can be created from the Bead Palette and adjusted with a corresponding frequency and intensity slider. The created Bead can be placed on the BeadPlayer timeline. Each track of the BeadPlayer represents an output contactor, with specific colour hue. The overview panel displays each page, while a page has duration of 1 s. The output arrangement visualization on the right column has interactivity; the user can drag to arrange the visualization icons to be in the same form as the output device.

Fig. 1.
figure 1

An overview of the Beadbox.

Full size image

3.1 Vibrotactile Notation

The notation design is the major component of the interface design. To build up a notation system, a basic information unit must be defined. The basic unit in this system is called a “Bead”. A Bead is developed to deliver essential information for vibrotactile art composition. It is similar to a “note” in music notation, providing composers with a mechanism to express and record their ideas. In order to devise a visualization of a Bead, the basic unit of the Emoti-chair output, a voicecoil, was used as an underlying metaphor as seen in Fig. 2. A Bead then is represented by two concentric circles where the outermost circle is described by darker, heavier line.

Fig. 2.
figure 2

Simple voice coil visual metaphor.

Full size image

Prior research has been carried out to investigate models of sensory substitution where different properties of perceptual systems can be used to represent various physical parameters such as frequency. These models were used to inform the representation of the vibrotactile parameters. The first mapping that has been explored was colour as a visual representation of auditory frequencies [14]. The second model was the relationship between the visual brightness of a colour and the audio frequency [15]. The researchers found that if a colour is brighter, then people tend to indicate that the sound stimuli would be higher pitch, and louder volume. If two variables have an overlapping information it is possible that users would be confused compared with a one-to-one mapping. Therefore, brightness was assigned to visualize the frequency level of vibrotactile stimuli. Figure 3 depicts a sample scale of the notation with different frequency level information.

Fig. 3.
figure 3

Colour brightness gradation (dark to light as frequency increases) by sample frequency level.

Full size image

Other researchers have found that there was a relationship between audio loudness and the size of a visual object. Louder sounds were seen to be related to the visual looming object by [16] where looming signals cause an avoidance response, resulting in a similar effect both in visual and auditory stimulations. For the Beadbox, the output intensity may be relative to the operating systems volume control (in logarithmic scale), or the size of output actuator. The Beadbox limits the maximum intensity to 50 dB to prevent any discomfort to the audience. The intensity coefficient scales linearly in the Beadbox from 0 to 100. The diameter of the inner circle of the Bead then represents amplitude whereby a smaller diameter would represent the lower intensity coefficient level. Figure 4 shows an example of the changes in diameter of the inner Bead circle showing lowering intensity coefficient.

Fig. 4.
figure 4

Sample Bead showing lower intensity coefficient.

Full size image

The last factor is timing information. To control the timing of the note, the user can drag and drop the Bead in the desired location along the timeline. The default duration of a Bead is set as 50 ms, which is the minimum stimulus duration a human needs to discriminate different frequencies [8]. From this point, the duration of each Bead can be controlled by connecting a start point to an end point. As seen in Figs. 5, 6 and 7, the pattern is interpreted as a single connected Bead which will be expressed without pause or cut. Duration can be assigned to two Beads on different tracks or two notes from different intensity or frequency levels. When there is a connection between different tracks, there will be constant decrement (or increment) at a linear rate on either frequency or intensity over time from the start Bead to the end Bead (Fig. 6). Similarly, a connection between two different intensities or frequencies will give a transition as well (Figs. 5 and 7).

Fig. 5.
figure 5

A Bead with a certain duration changing from a low frequency to high frequency.

Full size image
Fig. 6.
figure 6

A Bead with a certain duration changing from one track to another track

Full size image
Fig. 7.
figure 7

A Bead with certain duration from a low intensity to a high intensity.

Full size image

4 User Study

4.1 Methodology

A user study was carried out to evaluate the general usability of the Beadbox as a vibrotactile pattern authoring tool. Thirty people (20 female, 10 male) participated in the Beadbox user study. They ranged in ages between 18–64, with 19 in 18–24 age category, eight in 25–34 and one in each of the other three older age ranges (35–44, 45–54, 55+). Twenty-two participants were amateur artists; three participants were full-time professional artists; three participants were part-time professional artists; and two were non-artists. For types of creative practices, participants could select more than one specialization. There were four main categories of art practice: Group 1 = “Visual Artists: including Painting, Photography, Printmaking, Sculpture, Performing Art or Installation”, Group 2 = “None”, Group 3 = “Musical Artists: Music”, and Group 4 = “Combined art specialists: participants who specialize in multiple genres.” Six participants were in Group 1; six participants were in Group 2; five participants were in Group 3; and thirteen participants fell in Group 4.

For the study, participants completed a pre-study questionnaire, demonstration tutorial, a main composition session using Beadbox, and a post-study questionnaire. The pre-study questionnaire consisted of six forced-choice questions on demographics and artistic background of the participant. Basic instructions for the Beadbox were given in the 15 min demonstration where participants could practice, ask questions and discover the interface capabilities. In the main composition session, participants were asked to create a 5–10 s vibrotactile composition using the system. During the composition session, the participants were allowed to test and play their composition while sitting on the Emoti-chair. While participants were composing, the screen was being recorded, as was their verbal commentary. After the composition session ended, participants were asked to complete a post questionnaire. It consisted of ten questions from the System Usability Scale (SUS) [17], four questions regarding the interface and controls of each vibrotactile factor, and five open ended questions regarding their enjoyment of their composition when they played it on the Emoti-chair. The Likert-scale rating ranged from 1-strongly disagree to 5-strongly agree. We also asked what their intentions were for their composition, what were their expectations were, and what they liked and disliked about the Beadbox. Quantitative results were then analyzed with statistical methods and qualitative data with thematic analysis. Six themes were identified, defined and then subjected to a reliability test (see Table 2). Reliability was established by having two independent raters code about 20 % of the data randomly sampled, while including examples from each theme. The Intra-class Correlation statistic showed that there was agreement at a level of 0.84 and above. A single rater then completed the coding for the remaining data.

4.2 Results

4.2.1 Forced-Choice Questions from the Post-Study Questionnaire

A chi-square test was carried out for all forced-choice post-study questions to compare the participant responses to questions with chance. There was a significant difference between responses and chance for nine of seventeen questions (see Table 1 for all significant questions). The p-value was set at p < 0.05 and N was 30 for all questions. A Kruskal-Wallis non-parametric test was applied to the art practice groupings to determine whether there was a difference in the ratings between these groups. The result of the test showed that there was a significant difference in the complexity rating between the different artist specializations, X 2(3) = 7.94, p = 0.047, with a mean rank of 10.17 for Group 1, 22.25 for Group 2, 12.10 for Group 3, and 16.15 for Group 4.

Table 1. Results of chi-square test and descriptives for all significant Likert-scale type questions (p < 0.05). A rating of 1 is strongly disagree and 5 is strongly agree.
Full size table

4.2.2 Qualitative Data

All comments from participant’s responses to the open ended questions and audio/video recordings were coded into the defined themes (see Table 2). The number of occurrences for each theme is depicted in Fig. 8. A one-way analysis of variance (ANOVA) was used to find any significant differences between the themes for the number of comments participants made. There was a statistically significant difference between groups (F (8,111) = 3.166, p = 0.003). A post hoc Tukey’s HSD test was performed to determine the significant between the different pairs. There was a significant difference between the Interface Elements theme (M = 1.69; SD = 0.736) and the Creativity Positive theme (M = 1.00; SD = 0) (p < 0.05). There were no significant differences in other paired themes.

Table 2. Themes and definitions used for thematic analysis.
Full size table
Fig. 8.
figure 8

Frequency of comments from participants in each theme.

Full size image

5 Discussion

5.1 The Usability of Beadbox

The definition of usability according to the ISO/IEC 9126-1 standard is “the capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions” [18, p. 7]. This definition can be evaluated using three factors: learnability, functionality and its aesthetic. This discussion section focuses on these three aspects of the Beadbox. Responses to the three learnability questions (A.4, A.7, and A.10) showed that participants thought Beadbox was easy to learn. Given the relatively short amount of time participants were given to learn the concept of vibrotactile art (30 ± 10 min.), each participant learned how compose rapidly, and said that their composition was created as intended. According to Maguire [19], a usable system allows the user to concentrate on the task, which also means it is functional with reduced errors and reinforces learning for reduced training time. Some related comments mentioned:

“It was easy to learn how to use and flexible.”

“I like how it has a low learning curve for a relatively new art form”

For functionality, participants also agreed that the Beadbox was easy to use and that is was not unnecessarily complex. The notion of complexity, however, was related to the level of artistic knowledge and practice. Participants who specialized in some artistic practices thought that using the Beadbox was less complex than those who had no specialization. Although not significant, participants in visual art and music thought it was less complex than those from other fine arts, including performing art, and dance. In addition, fifteen out of forty-four comments in the Interface Elements theme related to Beadbox being easy to use. Participants’ responses to open-ended questions included that Beadbox was very easy to use, exemplified by:

“Very easy to use, instant results on what you’ve made.”

“I like how simple the program is to use, generally. It is very easy to understand and fairly intuitive. After a little trial and error, it was pretty easy to create a track.”

Participants also agreed that they wanted to use the Beadbox frequently and that the various Beadbox functions were well integrated. Out of eighteen comments, seven were listed under the Functionality Positive theme, and mentioned that the Beadbox was able to function so that each participant could create a vibrotactile art piece as they intended. Even though it was eventually achieved, some functions needed to be modified during the study cycles in order to address technical issues that arose during the studies. In addition to functionality, participants also made comments on the visual aspects of Beadbox. Thirteen of forty-four of the comments contained in the Interface Element theme were related to visual appeal. Participants liked that they could interact with a variety of simple shapes and colours. Participants said they really appreciated the colour arrangement and aesthetics, for example:

“The colour coding system for the intensity, different tracks, etc. was very helpful.”

“The colourful circles were most appealing to me.”

“It wasn’t solely for the aesthetic value, but an interactive piece that allows the participants to be part of an immersive space.”

Some participants mentioned that the circular-shaped design of a Bead and its simple layout on the Beadplayer improved learnability of the Beadbox. It also enabled multiple approaches to the compositional process. For example, participants who specialized in visual art commented that they focused more on the visual display of the Bead patterns rather than thinking about the vibration output. They tried to draw something using Beads, even after they knew that vibrations would result from those patterns. From the screen recordings and their composition piece, it seemed that they tried to draw a gun, stairs, a house or a smiley face. A future research direction could be exploration of the relationship between visually appealing compositions and their vibrational impact on audiences. Artists from different genres may be able to connect their specializations and the creation of vibrotactile art.

6 Conclusion

In this paper, a standalone vibrotactile composition tool, the “Beadbox” and its notation system was explained. The user study results on its usability showed that the tool is easy to learn and has good usability, with an attractive visual interface that is also functional. The Beadbox still can be improved by having options to provide different types of waveforms, or having secondary functionalities such as a detailed numeric frequency editor and a better navigation interface for the overview panel. The future research direction can be focused on further exploring the relationship between visual-tactile domains using the Beadbox as a drawing tool to output tactile stimulation.