Keywords

1 Background

Co-creation refers to creative activity that people who have different background and value share their dream and thought and achieve them together with allowing diversity of people [1]. This means that inclusive sense and cognition of togetherness are associated with embodied expression together called co-creative expression [2]. To explore the dynamics of co-creation, we have been focusing on practical method called “Hand contact improvisation” which is to contact others hands and embodied express impromptu together to achieve co-creation. Nishi, one of Authors of this paper, reported that hand contact improvisation have a facilitation function that deepens interpersonal relationship from individual state to togetherness state according to embodied expression changes shown in Fig. 1 [3, 4].

Fig. 1.
figure 1

Deepening process of interpersonal relationship in hand contact improvisation

Full size image

Furthermore, we have held workshops that various people regardless of developmentally disabled are attending in Ishinomaki and Higashimatsushima Miyagi prefecture, which are East Japanese great earthquake stricken cities. We found that hand contact improvisation is also useful for promote relationship with children who are developmentally disabled including autism spectrum disorder (ASD) which is characterized by tending to avoid interpersonal interaction. In our previous works, we developed interfaces to bring out embodied expression from developmentally disabled children for deepen interpersonal relationship with unpredictable others. This interface is cylindrical device that are used by holding both ends with others and embodied express together three dimensionally like hand contact improvisation [6]. Once we bring this interface to workshop, we observed that children who are developmentally disabled including ASD express hand contact improvisation using this interface shown in Fig. 2. However, it is difficult to find quantitative evaluation method about subjective interpersonal relationship from embodied expression yet [5]. In this research, we develop measurement interface toward establish quantitative evaluation method about subjective interpersonal relationship in hand contact improvisation from physical measurement data. In the beginning, we developed device for measuring load and orientation of interface in hand contact improvisation based on our previous works about single axis hand contact improvisation system [7, 8]. Also, we designed new sound media system with this interface toward deepening interpersonal relationship with unpredictable others by generating music from measurement data and present music from inner device.

Fig. 2.
figure 2

Co-creative expression interface at embodied expression

Full size image

2 Measurement Interface

2.1 Design Requirements

For evaluating interpersonal relationship through hand contact improvisation in three-dimensional movement, our interface needs to be able to measure load data of each hand and relationship of hand positions without interrupting embodied expression. We set design requirements of this interface below based on our previous work of single axis hand contact improvisation system.

  1. 1.

    The interface must have ability to measure load and orientation of hands.

  2. 2.

    The interface must be suitable size and weight for holding by two person’s hands.

  3. 3.

    The interface must not interrupt embodied expression by structure or mechanism of interface.

To meet these design requirements, new measurement interface needs to have measurement ability range from −20 to 20 kgf that is sufficient for measure hand contact improvisation hand load. Wireless communication system is also needed to prevent interruption to embodied expression and area of movement. Thus, we developed cylindrical measurement interface installing load cells and orientation sensor based on our previous interface design.

2.2 Overview of Measurement System

This interface is composed by three parts: frame part, sensor part, and record part. In a frame part, duralumin (A2017) is used as a fitting structure element due to the advantage for withstanding compression/tension loading and bending/torsional moment. In a sensor part, load cells (Minebea, CB17-3 K-11) are mounted to both ends of a frame part, and orientation sensor (Arduino nine axis motion) is installed to the center of a frame part. In addition to this, contact grip which is made by ABS are attached to load cell for holding them. In a record part, measurement data from a sensor part are recorded with 50 Hz sampling frequency by using wireless communication modules (Digi International, XBee S2B). In this way, we developed interface with 1146 g weight and 430 mm length and 110 mm width and range of load measurement from −20 to 20 kgf shown in Fig. 3.

Fig. 3.
figure 3

System overview of the measurement interface

Full size image

2.3 Performance Test

We measure three-dimensional position and load and orientation simultaneously using the measurement interface as a performance test. Test subject press and rotate the measurement interface alternately with 60 bpm metronome sounds. Results of the test performance are shown in Fig. 4(c). This figure shows three-dimensional position, load, and orientation defined by Fig. 4(b) of the interface. We can see that pressing and rotation are detected by interface based on the order in Fig. 4(a) from the measurement results. This result suggests that the interface have ability to measure load and orientation of itself simultaneously.

Fig. 4.
figure 4

Performance test of the measurement interface

Full size image

3 Sound Media System

3.1 Design of Sound Media System

To bring out diverse embodied expression and deepen interpersonal relationship, we designed new sound media system using this interface shown in Fig. 5. This media system is composed by three parts: measurement part, sound media generation part, sound presentation part. Measurement part achieves load and orientation data using same system as measurement interface. Sound media generation part produce sound media with sound processing software (Max/MSP) in real-time by using achieved data from measurement part. Sound media presenting part have Bluetooth wireless speaker installed in the center of interface and the speaker presents sound media. This structure make music sound source position move according to embodied expression movement. Method of generating sound media is shown below.

Fig. 5.
figure 5

System overview of sound media system for hand contact improvisation

Full size image

3.2 Method of Sound Media

Music of sound media system needs to be changed by body movement and load of hands in hand contact improvisation. Music of sound media also need to be diverse and having unpredictability. This time, we focused on phase music that is famous for Minimal music of Steve Reich works. Phase music uses repetitive melody lines and phase difference for generating diverse music from simple rules. We designed phase music system using load and acceleration data for manipulating volume of sound and phase difference. This sound media is mainly generated by three processes: setting melody line, manipulation of phases, presenting sound. Sound generation processes are shown below.

(a) Setting Melody Line

Melody line in this media system is data for setting sounds tone and timing. Timing of sounds is described by set of numbers from 0 to \( 2\uppi \). For simplification of system, we constrain sound tone and used set of numbers which are randomly chosen from \( 2\uppi \) divided by 12 as sounds timing.

(b) Manipulation of Phases

To play sound using melody line set by process (a), we used parameters called phases that have ranges from 0 to \( 2\uppi \). To make change in music from phase music, phases are needed to be manipulated toward both synchronous state which has small difference of phase difference and asynchronous state which has large phase difference. For making these states, we manipulated phases using phase oscillator model described below [9].

$$ \mathop {\theta_{i} }\limits^{ \cdot } = \omega_{i} + \frac{K}{N}\sum\nolimits_{j = 1}^{N} {\sin \left( {\theta_{j} - \theta_{i} } \right),\;i = 1 \ldots N} $$
(1)

where \( \theta \) is phase. Change rate of \( \theta \) is defined by natural oscillation frequency \( \omega \) and member of distance between phases and a coefficient K that decides effect of entrainment. In this system, two phases are used and natural oscillation frequency \( \omega \) is set near to \( \uppi \). Coefficient K is manipulated by load data to make synchronous and asynchronous states.

(c) Presenting Sound

System present sound by using timing set in process (a) and phase value manipulated (b) method. When phase value passes through timing value, arbitrary sounds are presented from speaker in the center of the interface. Volume of sound media also changed by length of acceleration vector achieved from measurement data for adding effect of movement in embodied expression. In this way, we achieved to design sound media system that has changed by load and acceleration of measurement interface in hand contact improvisation.

4 Results and Discussion

Measurement interface are used for hand contact improvisation in a range of 2.0 m width and 2.0 m depth and 2.5 m height with 90 s experiment time by two subjects. We measured three-dimensional position and load and orientation of the measurement interface and hip position of subjects simultaneously. Results are classified by an expert of embodied expression to condition of co-creation and condition of not co-creation. Figure 6(a) shows three-dimensional position and orientation data form measurement data. To evaluate movement of interface, we plot return map of local maximum value and local minimum value about back and forth position and vertical position of the measurement interface in Fig. 6(b). We can observe that distribution of return map in co-creative condition is larger than not co-creative one. Figure 7 shows load data and average value and difference value. From a perspective of fluctuation range, we can find differences between co-creative condition and not co-creative condition in both load average and difference. These results suggest that this new measurement interface have an ability to evaluate interpersonal relationship through hand contact improvisation.

Fig. 6.
figure 6

Profiles of movement data in hand contact improvisation

Full size image
Fig. 7.
figure 7

Profiles of stress data achieved from the measurement interface

Full size image

We also conducted experiments about sound media with the interface. The interface is used with 90 s embodied expression comparing situation of using sound media and situation of not using sound media. Results of experiments are shown in Fig. 8. We can see increase tendency in peak number more than 3.0 kgf of load data by using sound media.

Fig. 8.
figure 8

Profiles of movement and stress data using the interface with sound media system

Full size image

Furthermore, once we bring the interface to workshops or practical fields, we observed that developmentally disabled children who are handicapped with interpersonal communication skills interact with others using the interface presenting sound media shown in Fig. 9. These finding suggests that this sound media system have an ability to deepen interpersonal relationship in Fig. 1. Authors think that this interface has facilitation effect toward co-creative expression.

Fig. 9.
figure 9

Situation of embodied expression with the interface in workshops

Full size image

5 Conclusion

In this research, we developed new measurement interface focused on hand contact improvisation for evaluating interpersonal relationship with others toward understanding dynamics of co-creative expression. The measurement interface is cylindrical device that have ability to measure load and orientation of hands. We also developed sound media system which represents hand contact improvisation for deepening interpersonal relationship with others. We conducted experiments about hand contact improvisation using the measurement interface and results showed that differences between co-creative state and not co-creative state in load data. In addition to this, we observed that sound media system with the measurement interface is used for embodied expression by children who are developmentally disabled including ASD children. These results suggest that our new design method of interface is not only useful for evaluating interpersonal relationship through hand contact improvisation but also have function of facilitation that brings out diverse embodied expression toward co-creative expression.