Audemes at work: Investigating features of non-speech sounds to maximize content recognition

Abstract

To access interactive systems, blind users can leverage their auditory senses by using non-speech sounds. The structure of existing non-speech sounds, however, is geared toward conveying atomic operations at the user interface (e.g., opening a file) rather than evoking broader, theme-based content typical of educational material (e.g., an historical event). To address this problem, we investigate audemes, a new category of non-speech sounds whose semiotic structure and flexibility open new horizons for the aural interaction with content-rich applications. Three experiments with blind participants examined the attributes of an audeme that most facilitate the accurate recognition of their meaning. A sequential concatenation of different sound types (music, sound effect) yielded the highest meaning recognition, whereas an overlapping arrangement of sounds of the same type (music, music) yielded the lowest meaning recognition. We discuss seven guidelines to design well-formed audemes.

Introduction

Users interact with computer interfaces through different sensory modalities. Traditionally, the main modality has been visual, through the graphical user interface (GUI). Although touchscreens and haptic interactions have gained wider acceptance and have begun to significantly impact interface design, these primarily supplement the visual modality, which remains dominant. Walker et al. (2006), however, argue that with the increasing number of mobile devices, in which the screen real estate is significantly reduced, the auditory channel has begun to assume much greater importance. Brewster et al. (1993) point out the potential of non-speech sounds to reduce the stress on the user to focus attention on smaller or non-stationary visual targets. Additionally, non-speech sounds can be used complementarily to the visual output to increase the quantity of information conveyed to the user. Many other researchers have acknowledged the relevance of sound in user interfaces and investigated ways to utilize this modality (Brewster, 1994, Brewster et al., 1995a, Conversy, 1998, Edwards, 1989, Kantowitz and Sorkin, 1983, Sanders and McCormick, 1987).

Non-speech sounds have been used either to complement visual interfaces by adding another information channel, or used as the main channel when dealing with the visually impaired community, or used in “eyes free” context applications, such as driving. In these contexts, sounds have served as a valuable asset to communicate information where the visual channel was limited. Also, studies advocate that sound can be an influential cue for memory enhancement (Sánchez and Flores, 2003), and researchers have used sound to teach blind children math by communicating the formal traits of the formulas, such as its length and complexity (Sánchez and Flores, 2005, Stevens et al., 1994). The use of non-speech sounds in these different circumstances, suggests the increasing importance of non-speech sounds and efforts to perfect their communication abilities through user interfaces.

Despite the recognized importance of non-speech sounds, their design has been mainly conducted in an ad hoc manner, rather than following any guidelines. The lack of guidelines for designing non-speech sounds for interfaces has been noted by Back and Des (1996), who maintain that sounds should be designed as narratives or stories. Mustonen (2008) and Pirhonen et al. (2007) have noted this problem and suggested devising a theoretical framework for non-speech sound design, which can be used in auditory displays. Brewster et al. (1995a) have empirically derived design guidelines for a specific type of non-speech sound, but those are not comprehensive to the various non-speech sounds that exist.

Non-speech sounds have been the primary interest of our research for five years. In a previous study (Mannheimer et al., 2009), we developed audemes, which are short non-speech sound symbols, comprising various combinations of sound effect and music sounds. Audemes are a type of non-speech sounds whose meaning is derived from the combination of sounds. For example, combining sounds of a “key jangle,” which is used to represent keys, and a “car engine starting,” used to represent “a car,” generates the meaning of “driving.” Further, if we add a sound of “seagulls and surfing,” the final audeme could mean “vacation and trip to the beach.”

Just as the other designers of non-speech sounds, we have also enjoyed a wide freedom to explore various constructive and semantic strategies in generating audemes. However, we maintain that the time is ripe to consider formulating guidelines to help researchers and application designers achieve a more standardized approach to design non-speech sounds, one that could be more easily learned and applied in the everyday practice. To achieve this goal, we conducted three experiments with blind and visually impaired participants in which we examined the optimal combination of audeme attributes that can be used to facilitate accurate recognition of audeme meanings. This paper reports the creation of seven basic guidelines that can be used to design well-formed audemes.

The following research questions are central to this study:

Q1: How well do audemes aid in recognizing information?

The goal of this question is to establish the level of effectiveness different types of audemes have in terms of recognizing any textual content that is associated with them. Once this is established, it is important to determine the time period of how long an audeme can be effective, so that reinforcement can be applied to maintain the effective link between the audeme and the content it is representing.

Q2: Which combination of audemes is the best for accurate information recognition?

Audemes are created from different attributes (music or sound effects, serial or parallel arrangement, and so on) so it is important to understand which combinations are the most effective for recognizing the content associated with them.

The outcome of these research questions will help us better understand the nature of audemes along with their strengths and weaknesses. Moreover, it will help derive a set of initial guidelines for designing effective audemes to be used by acoustic interface designers.

The remainder of the paper is organized as follows. Section 2 will review related work that focused on models of learning using sounds, the meaning sounds generate based on modes of listening theory, and the different types of sounds in Human–Computer Interaction. Section 3 introduces three experiments conducted to generate guidelines for creating effective audemes. Section 4 highlights synopsis of findings from the three experiments. Section 5 describes the discussion and limitations to the three experiments along with hints to future work. Finally, Section 6 describes the concluding remarks.