Keywords

1 Introduction

Creative people are unrestrained, appear to lack discipline, and display a great deal of curiosity about many things. Children, to some degree, are the embodiment of creativity [1, 2]. To think of innovative ideas and solutions, there are tools and techniques that creativity experts use to help people think differently, and problem-solve more creatively. We believe that Tangible User Interfaces (TUIs) may provide new approaches to support creativity.

TUIs are a type of human computer interaction design based on graspable physical objects that are shifting the actions required for interacting with digital information from pointing and clicking to holding, grasping and moving physical objects. TUIs are the coupling of physical objects and digital information, and eliminate the distinction between input and output devices, such as mouse and display [3, 4]. For example, Fig. 1 illustrates SifteoTM cubes, a type of TUI, and Fig. 2 illustrates children using the cubes.

Fig. 1.
figure 1

SifteoTM tangible user interface cubes

Full size image
Fig. 2.
figure 2

Experimental design left: cubes condition/right: poster condition

Full size image

Tangible interaction takes advantage of how people typically interact with physical objects in the world and brings those affordances to interactions with digital environments. Recent studies of TUIs, and physical objects more generally, and creative thinking have lead us to explore two hypotheses: 1. Tangible interaction increases the quantity of creative ideas. 2. Tangible interaction encourages the development of creative concepts.

TUIs have been shown to affect designers’ cognition during a design task [5, 6]. Kim and Maher [7] compared TUI and GUI on a floor plan configuration task: They found an increase in epistemic actions and through a protocol analysis were able to observe an increase in the cognitive processes typically associated with creative design. The affordances of TUIs such as manipulability for physical (re-)arrangements may off load cognition to the tangible objects and reduce cognitive load associated with spatial reasoning. Brereton and McGarry [8] studied the role of objects in supporting design thinking as a precursor to designing tangible interaction. They found that design thinking is coincident with gesturing with objects and recommend that the design of tangible devices should consider a tradeoff between exploiting the ambiguous and varied affordances of specific physical objects. If TUIs offer greater opportunity for epistemic actions, then they may improve creativity by affording creative exploration through physical action.

Most studies of TUIs have been undertaken from a HCI technology viewpoint, which aims to describe fundamental technical issues and implement prototypes. Typically, initial user studies are conducted on prototypes focused on functionality only. Prototypes have not been evaluated from a cognitive perspective, which can be used to guide the development of new technology and applications. While many researchers have argued that TUIs improve spatial cognition, there has been no empirical evidence to support this [5, 6]. Although some researchers have reported on the users’ perception of TUIs using survey questionnaires or designer comments, the subjective nature of self-reports calls into question their validity as measures of cognitive ability [9]. Technology-oriented studies, anecdotal views and subjective measurement are also insufficient as measures of cognitive ability leaving a gap in our knowledge. Our research addresses this gap on the effect of TUIs on cognition by reporting on an experiment design and coding scheme for comparing design cognition in two conditions that are distinguished by the affordances of pointing and grasping.

There is evidence that gesturing aids thinking. There is extensive evidence that gesturing with our hands promotes learning [10, 16], and aids problem solving [11], but few studies have explored actions with objects [12], and none have compared tangible object and intangible interaction. When children are learning to count, the learning is facilitated by touching physical objects [13, 14]. Kessell and Tversky [11] show that when people are solving and explaining spatial insight problems, gesturing facilitates finding solutions. Goldin-Meadow et al. [15] found that children who were instructed to imitate a teacher’s gestures learned a strategy for solving math problems compared with children who did not gesture. Goldin-Meadow and Beilock [17] summarize these and related findings as “gesture influences thought by linking it to action” (p. 667), and “producing gesture changes thought” and can “create new knowledge” (p. 668). These studies show that gesture, while originally associated with communication, is also related to thinking. Tangible interaction design creates an environment that encourages actions on objects and therefore induces more gestures and actions than traditional GUIs. This paper provides an experiment design that can be used on learning tasks for a better understanding of the impact of tangible interaction on learning.

Studying creative processes provides insight into creative cognition. The processes for generating potentially creative solutions are described generally by Boden [18] as: combination, exploration, and transformation where each one is described in terms of the way in which a conceptual space provides a basis for producing a creative solution and how the conceptual space changes as a result of the creative solution. Gero [19] describes processes for generating potentially creative designs, including combination, transformation, analogy, emergence, and first principles. These processes explore, expand or transform the relevant conceptual space. Maher et al. [20, 21] identify characteristics of creative products as being new, surprising and valuable. These characterizations of creative processes and products provide a basis for comparing the effect of TUIs and GUIs. In this paper, we use the concepts of new and surprising to compare two conditions: pointing and grasping.

In this paper, we describe an experiment design [20] and a coding scheme to measure how graspable tangible devices differs from pointing in a creative task of combining words and giving the combination a meaning. Ultimately, our goal is to study how interfaces based on physical objects (i.e., TUIs) engage human cognition differently than traditional computer interfaces that do not include grasping within a design context. In order to see how TUIs change cognition, we segmented and coded the verbal protocol data using the function-behavior-structure coding scheme [22] to compare the participants’ references to design issues across the two conditions. We show how the two conditions differ in the number of new and surprising concepts introduced as well as the characteristic design reasoning processes in the two conditions.

2 Experiment Design for Studying the Effect of TUI on Cognition

This section describes an experiment design for comparing TUI, with a focus on the grasping affordance, and GUI, with a focus on the pointing affordance. The experimental task is conceptual combination as a design task, where the design task is a synthesis of prescribed components (words) and the creation of a meaning for selected combinations. We adapted a conceptual combination task based on Wisniewski and Gentner [23] using nouns that varied semantic dimensions: natural kinds/artifacts (e.g., frog/box) mass/count nouns (clay/candy). The participants performed the task in two conditions: a poster condition and a cubes condition. We carried out a protocol analysis: we collected audio/video data while the participants were engaged in the task, then coded and analyzed the data. We expected that rearranging cubes would generate more word combinations and thus more creative meanings than reading words on a poster. We also expected that grasping cubes would engage spatial cognition and spatial metaphors more than reading words from a poster.

2.1 Experimental Environment

In the experiment, the participants are asked to combine words from a given set of 6 words, and then describe meanings for the combined words. In the instructions, the participants are asked to think about the function, behavior, and structure of the combined word when creating its meaning by giving them examples of function (what is it for?), behavior (how does it work?) and structure (what it is made of or look like?). The visual display features of the word stimuli were similar in font, size, and layout in a square border, and we varied whether words were displayed on tangible user interface cubes or printed on a poster board. In the Cubes Condition (Fig. 2, Left) words were graspable and rearrangeable because each word was displayed on a tangible cube. In the Poster Condition (Fig. 2, Right) words were printed on a poster board, and thus intangible.

In pilot studies with words displayed on computer tablets we found that even when participants were instructed that words could not be moved they tried to manipulate them by attempting dragging and tapping gestures and grasping the tablet. We designed the cubes and poster conditions to avoid these potentially confounding variables and to limit the change in the two conditions to varying human interaction affordances while maintaining the same visual features of the word stimuli across conditions. Participants could point at the poster in the poster condition and pick up cubes in the cubes condition while reading word stimuli and speaking their responses.

2.2 Participants

Forty 6th grade children (aged 11–12) participated in the experiment. This number of participants provided data for 20 pairs of participants. We chose 6th grade children because of their more developed ability to compose creative meanings while engaging them in a task that would be received as age-appropriate, which was supported by the middle school’s faculty. Due to procedural errors we were able to use the data from 14 of the 20 pairs of participants.

2.3 Words as Design Elements

Two sets of words were constructed to serve as stimuli for the word combination task, designed to form compounds, which would promote creative definitions. The two word sets consisted of six words each, matched within and between sets on a variety of psycholinguistic properties so they could serve as counterbalanced stimulus sets in both experimental conditions, while minimizing potential psycholinguistic effects between conditions unrelated to our hypotheses. All words were one syllable nouns representing concrete basic-level category objects. Each noun represented one of six disparate semantic categories: food, furniture, tool, clothes, vehicle, and animal. This prevented word combinations from forming same-category meanings, thus promoting creative thinking (Table 1).

Table 1. Word set stimuli by semantic category
Full size table

Six Sifteo cubes were programmed to display one word per cube. The display did not change, and no other cube sensors or capabilities were active. The display screen of Sifteo cubes is housed in a square frame with rounded corners. We designed printed poster words to match cube displays on task relevant perceptual attributes: text font and size; words appeared centered in a rounded square; and initial spatial arrangement of the cubes and printed squares matched. Each cube displayed the same word throughout the session. We printed words on a single poster paper (Fig. 2 Right) after pilot studies explored alternatives, such as sticky notes, and found that a poster board secured to a table served best to match the appearance position and reach of the cubes (Fig. 2 Left). Photo images of cubes were not used to avoid reminding participants of the graspable affordance of real cubes. Two sets of six cubes each were assigned to display words from Word Set 1 and Word Set 2, respectively. Two poster papers were printed containing words from Word Set 1 and Word Set 2.

2.4 Experimental Procedure

The experimental design was within-subjects consisting of an instruction phase followed by two experiment conditions: Poster Condition and Cubes Condition in two counterbalanced blocks. The two Word Sets were counterbalanced by condition type (Poster, Cubes), block order (Block 1, Block 2), and WordSet (Set 1, Set 2).

Participants were assigned to 20 pair groups. Based on pilot studies, working in pairs promoted talking and gesturing, while maintaining focus on the task, because participants had a peer co-engaged in the task with whom to verbalize their responses. The duration of the experiment was approximately 20 min consisting of the instruction phase followed by two experimental blocks of 5 min each. During the instruction phase two participant pairs sat together in a common room, afterward each pair of participants moved to an assigned testing room. Participants were instructed to “combine words and come up with as many creative meanings as you can”. The task was presented as a game to encourage the children to explore many word combinations, and verbally describe creative meanings. Instruction consisted of an example: “the word fish and the word car are things everyone knows about, but nobody knows about a fish car”. Three questions based on function, structure, and behavior [22] encouraged creative thinking: Who can tell me what a fish car might look like? —what a fish car is for, or what it does? —how a fish car works?

In the experimental phase each participant pair sat at a table with a poster paper (Poster Condition) or Sifteo cubes (Cubes Condition). Participants self-selected their choices of word combinations and how they took turns presenting their creative ideas to their paired partner. Experimental sessions were video and audio recorded.

3 Data Analysis

3.1 Segmentation and Coding

Our analysis of the video stream for each session involves segmenting the video into discrete elements defined by a start time and end time, and assigning a code to each segment. We started by segmenting the verbal stream according to speaker, and then segmenting into smaller segments so that a segment is formed around the utterance of a word combination or around the definition of a word combination. To analyze the participants’ consideration of design issues, we had an additional stage of segmentation in which each segment is associated with one “FBS” code using the FBS coding scheme described below. This final segmentation and FBS coding were done simultaneously. Coding was conducted by four of the authors (Maher, Gero, Lee, and Yu) to ensure agreement. Then a single coder coded all sessions twice, separated by a period of several days, followed by an arbitration process to identify and resolve differences in coding. This coding process ensures a uniform coding across all 20 sessions.

3.2 FBS Coding Scheme

The Function-Behavior-Structure (FBS) coding scheme used is an adaptation of the one presented in Gero [22]. We mapped the word combination task onto the components of a design task: the given words are effectively the design requirements and serve as the building blocks for the participants’ designs. By asking the participants to create a meaning for word combinations, they are asked to describe the design that results when you put two or more words together. For example: bee shirt, when they talk about what a bee shirt is they talk about what it looks like, how it behaves, and what it is for. This allows us to code each meaning for a word combination as if it were a design description and can be labeled as F, B, or S. Specifically, our FBS codes are:

  • R: Requirements. This is when the participant makes a verbal reference to one of the six words printed on the poster or cubes.

  • F: Function. This is when the participant talks about the purpose, use, or function when describing the meaning of the word combination.

  • Be: Expected behavior. This is when the participant talks about an expected behavior in the meaning of the word combination.

  • Bs: Behavior from structure. This is when the participant talks about whether the structure in the meaning of the word combination can actually achieve the expected behavior.

  • S: Structure. This is when the participant talks about the appearance, the form, the spatial qualities, and the material properties of the meaning of the word combination.

  • O: Other. This is used when a participant repeats a phrase or talks about something that is not relevant to the task.

3.3 New and Surprising

A definition of creativity [24] may focus on novelty as the primary criterion and claim that novelty is expressed as a new description, new value, or a surprising feature of a creative product. Alternatively, many definitions will state that value is the umbrella criteria and novelty, quality, surprise, typicality, and others are ways in which we characterize value for creative artifacts. Maher [21] presents an argument for three essential characteristics of a product to be considered creative: novelty, value, and surprise.

Amabile [25] introduces a Consensual Assessment Technique (CAT) in which creativity is assessed by a group of judges that are knowledgeable of the field. Within this technique, Amabile defines a cluster of features associated with creativity for the judges to rate that are specific to the artistic or verbal artifact being assessed (for example, in an artwork: creativity, novel idea, variations in shapes, complexity, detail). The CAT does not assist in developing a common set of metrics for evaluating creativity but instead provides a common technique for people to judge creativity.

To compare the creativity of the descriptions of the word combinations, we coded when each F, B, or S segment introduces a new and/or surprising idea for that pair of participants. The segment was coded as New if a word in the segment had not been used before in the session. The meaning of surprising is derived from the distinction between novel and surprising in Maher [21]. Surprising ideas are those not normally associated with the function, behavior or structure of the words in the word combination. If a response is possible, viable, realistic, feasible, or if it makes sense, we did not code it as a surprising word. A segment was coded as Surprising if it contained words that introduce unique concepts, different from the inherent function of the requirement words.

For example, when explaining chair egg in session with student pair 3 (P03), Child 2 said “a chair and an egg what if there is like a whole cracked up egg and you can just sleep on it like a vampire or just like be in the egg.” A vampire in this context is unexpected and was coded as surprising.

4 Results and Analyses

4.1 Verbal Response

As participants composed word combinations and described meanings they interacted with the stimulus materials by pointing to and touching words on the poster, and by grasping, arranging, and combining the cubes. The length of combinations ranged from two to six words. All pairs of participants followed a two stage response pattern that repeated throughout the session: Search then Description. In the Search phase, the participants verbalized a series of word combinations as they searched for one to describe. In the Describe phase, the participants talked about the meaning of one of the word combinations identified in the Search phase. The two phases repeated until the session ended. Unexpectedly, some participants’ initial search responses included additional familiarization with the materials.

4.2 FBS Results

The overall statistical analysis of the FBS coding across all participants is shown in Fig. 3. Figure 3 shows the average of the percentages of the number of segments for each of the FBS codes, with errors bars showing the standard deviation. This analysis shows that there is no significant difference in the average of the design issues across the two conditions.

Fig. 3.
figure 3

Average of the percentage of segments for the five FBS codes listed by their design issue labels in the two conditions: poster and cubes.

Full size image

We compared the number of Search phase segments and the Description phase segments in the two conditions for each pair of participants. For the Search phase we compared the percentage of segments coded as Requirements. For the Description phase we compared the percentage of the sum of the segments coded as F, Be, Bs, and S. For this initial pair-wise analysis we labeled an increase (or decrease) of more than 10 % as an increase (or decrease). Table 2 shows the results of this analysis. The results show that in the Cubes condition 11 of the 14 pairs of participants show an increase in segments coded as Requirements, which we associate with the Search Phase. The results show that in the Poster condition 13 of the 14 pairs of participants show an increase in segments coded as F, B, and S, which we associate with the Description Phase. One possible interpretation is the affordances of the cubes had a positive effect on the number of alternative word combinations considered and resulted in less talking while the participants were describing the meaning of the word combinations. Alternatively, affordances in the Cubes may have negatively influenced the Search phase, or affordances in the Poster may have positively influenced the Description phase. These data cannot distinguish among these possible interpretations.

Table 2. Percentage of number of segments in Search ® and Description (F, B-E, F-S, S) for each pair of participants
Full size table

4.3 New and Surprising Results

New and surprising ideas are recognized when a new attribute is encountered in a solution, a previously unknown value for an attribute is added, or a sufficiently different combination of attributes is encountered [19]. The participants that produced the most new and surprising ideas displayed a great deal of curiosity about many things and offered unusual, unique descriptions of the meanings of the word combinations.

We counted and compared each pair for an increase or decrease in the number of segments that were coded New or Surprising for each condition, cubes and poster, shown in Table 3. In the cubes condition, 6 out of 14 pairs had more New segments and 11 of the 14 pairs had more Surprising segments. However, the total number of New and Surprising segments showed minimal difference and therefore these are not distinguishing results. The mean value of New segments for the poster condition was 48.4 and the cubes condition was 41.0. The mean value of the Surprising segments for the poster condition was 11.4 and the cubes condition was 12.0. These results indicate that quantity of New and Surprising segment did not differ by condition type.

Table 3. Number of new and surprising segments for each pair of participants
Full size table

In this study we assumed that using new words instead of commonly used words in explaining the meaning of a word would lead to surprising or creative meanings and looked into relationship between New and Surprising segments. We found that more New segments does not necessarily mean more Surprising segments. Although more New words are uttered in the poster condition of P06, P11, P13, P17, P18 and P20, for these pairs, the cube condition drew out more novel and surprising meanings.

As we noted in Sect. 4.2, 13 of the 14 pairs had more total Description segments in the poster condition. However more total Description segments did not necessarily mean they used more new words or created more surprising meanings. In spite of the fact that total Description segments is higher in the poster condition, we found that P02 and P10 used more New words in the cube condition and P01, P02, P06, P11, P13, P17, P18 and P20 drew out more Surprising meanings in cube condition. The difference in the number of segments is not large, but our results show that the number of segments in the Description phase does not relate positively with the number of New or Surprising words. The indication is that even with the fewer words used to describe the meanings in the cubes sessions, the participants had a higher incidence of New and Surprising words.

5 Discussion

The association of gesture with thinking leads to the possibility that interaction modalities that encourage gesture, and more generally body movement, may affect cognition. In this paper we present an experiment design that compares grasping and touching in a word combination task to explore the affect of tangible interaction on creative cognition. The experiment design isolates pointing and touching through the design of experimental materials that provide the affordance of pointing in the Poster condition and the affordance of grasping in the Cubes condition. In an analysis reported in Maher et al. [26], we analyze pointing, grasping, and gesture in the data collected in these two conditions and show that there is a significant difference in body movement in the two conditions, beyond the simple difference between pointing and grasping. In this paper, we analyzed the same data to compare the two conditions using the FBS coding scheme and an analysis of new and surprising concepts.

Our results show that the two conditions show a difference in the number of segments for Searching (Cubes condition shows increase in segments for searching) and Description (Poster condition shows increase in segments for description). We believe this may be due to the affect of grasping the objects in a TUI, when compared to pointing in a GUI-like session. We also noticed that while there were more segments for the Description phase in the Poster condition, this did not correspond to an increase in the number of new and surprising ideas.

This paper provides an experiment design and coding scheme that is a baseline for future studies of the impact of tangible user interfaces on design and creative cognition. In the future we will carry out additional experiments to determine if a task that requires spatial reasoning (for example configuring blocks) or learning will show a stronger effect on design and creative thinking.