StringWeaver: Research on a Framework with an Alterable Physical Interface for Generative Art

Abstract

In order to improve the input interface for generative art, the author was inspired by a traditional game called string figure to design a framework with an alterable physical input interface named StringWeaver. The input system of StringWeaver is consisted of strings (made of black conductive rubber tube) which can be customized physically by rearranging and blob tracking system that can track audience finger. The visual output is directly projected on the input interface with music and sound generated. StringWeaver was proved to be useful by three prototypes developed under the framework of it. Limitations and future work are introduced at the end of the paper.

1 Background

In recent years, there has been much active research on generative art. Contemporary research on generative art mainly focuses on several different levels, such as its history, principles, implications, its relationship with art and artists, ways of creation, fields of application, aesthetic evaluation and so forth. Among these levels, the studies on the ways of creating generative art put more emphasis on how to control the content generated by computers. Methods likes Genetic Algorithm、L-system、Catastrophe Theory (from René Thom)、Cellular Automaton、Fractal Generation and Chaos Simulation were brought forward, while the human factors were ignored. Generative art was born and used, according to Professor Philip Galante’s theory,¹ long before computers were invented. Famous examples include: W.A. Mozart, a famous musician, obtained inspirations for his composition by throwing dices into numbered grids; Ellsworth Kelly, an American artist, made randomly colorized collages with children’s books; William Burroughs, an American writer, created visual artworks by blasting paint with shotguns. Generative art was born in the physical world with physical operations, but when computer age came, physical operations were gradually forsaken and artists turned to regulated interfaces of computers, e.g. keyboards and mice. For art creation, however, keyboards and mice are so non-intuitive that they cannot encourage artists to release their passion and inspirations. Is there a more intuitive computer interface for art creation? The answer is yes.

Physical interface (PI in short) is also known as tangible media with roots in seminal work of Wellnerand Fitzmaurice, which uses physical forms to represent data. Instead of a generic screen, mouse, and keyboard capable of representing all types of data, PI uses specific physical forms to represent and manipulate the pieces of data in the system. PI often uses simple, transparent mechanical structures, so the user can use his or her existing knowledge of the physical world to figure out how to operate them and what they mean. A concept underlying these interaction styles is to build on the “natural” equipment and skills humans have acquired through evolution and experience and exploit these for communicating with the machine [2].

There are already many outcomes from PI research. It has been prototyped in several different areas. Computer will continue to play an important role in this field, but the constraints brought by its rigid input devices, such as keyboards and mice, must be broken to meet the tendencies of modern times. On the basis of PI research outcomes, this paper tries to put forward a framework for the creation of generative art using physical interfaces, so as to meet the requirements to create certain kinds of generative art, and to allow artists to create in more artistic, intuitive and friendly ways.

2 Related Work

Although different kinds of PI research were mostly done for pragmatic usages instead of artistic creation, it’s difficult to differentiate art from practical design strictly. A small fraction of research projects even began to explore possible relations between physical interfaces and generative art. For this reason, the perspectives of much research are highly valuable as references. The following paragraphs include analysis of some typical cases and their contexts, from which the research direction of this paper was defined.

Project PY-ROM (2009) was a matchstick-like video recording and storage device concept that burned itself away after being used. Reference [3] This project was just a concept instead of an actualized product. This concept enabled users to trigger a virtual event, recording and playing a video clip, via an action uniquely bound to match sticks: striking. Project Cord Uis (2015) used sensorial augmented cords that allowed for simple metaphor-rich interactions to interface with their connected devices. Reference [4] A user might control brightness, direction and other properties of the light by twisting and pressing the cord as if it was a water hose. The design thinking behind these two projects was similar: the utilization of simple daily objects and the using habits behind them.

Project Wavo (Fig. 1, 2010) was an interactive ball to express light waves with wave equation. Reference [5] A user might appreciate the beauty of wave equations with this device, and might also wave it to see the generation of different patterns. Project Trackmate (Fig. 1, 2009) was a customizable tangible system that allowed a computer to recognize (track) tagged objects and their corresponding position, rotation, and color information when placed on a surface. Reference [6] A user might move, rotate colorful cubes to control the changing shapes on the screen. The design thinking behind these two projects was: transplanting a person’s basic cognition and intrinsic manipulation, such as moving and rotating, of physical objects to a tangible agent. The difference between them is that: in Project Wavo, the agent was a single man-made geometry in fixed shape, while in Project Trackmate, the agent was an alterable set of physical objects that might be redefined and combined according to certain rules.

Fig. 1.

Project Wavo (left) and project Trackmate (right)

Project Andante (Fig. 2, 2014) visualized animated characters walking along the piano keyboard that appeared to be playing the physical keys with each step. Reference [7] An ordinary user might simply play with Andante for fun or for learning, and as for advanced users they might use their skills of playing a piano to make Andante’s visual effects even more interesting and exciting. Project BiiliARt (Fig. 2, 2014) was a framework for processing visual and auditory textures in an augmented billiards environment that enabled real time artistic creativity without imposing predefined interaction rules or constraints. Reference [8] Users might use their skills of playing pool to interact with it. In fact, Andante and BiiliARt were both complex systems. To make them simple for end users, the key point was the props with embedded systems, piano and pool table. They transplanted objects or systems in the real world together with the complex behaviors behind them, rather than an instinctive action.

Fig. 2.

Project Andante (left) and project BiiliARt (right)

Based on the analysis above, various input modules are organized into a diagram as shown in Fig. 3, where X axis stands for physical appearance of input interfaces, Y axis stands for interaction scheme. The essence of this diagram is to relate interaction schemes (Y axis) to the input interfaces (X axis), and thus to clearly reveal the design rules of physical input interfaces. In this diagram, the grey areas either lack physical interfaces or natural interaction methods. Although most of the present approaches of generative art creation belong to these grey areas, it’s what needs to be changed. On the Y axis, the difference between simple habitual manipulation and behavior pattern needs a second thought. If a simple habitual manipulation is mapped onto a fixed physical object, the resulting control mode is monotone or very limited, as in project Wavo or PY-ROM. Contrarily, if a behavior pattern, as in project Andnate, is mapped onto an alterable physical agent, as in project Trackmate, very rich control can be expected. It is evident that optimal interfaces for creating generative artworks shall provide rich controls, which graphically means that only green areas in Fig. 3 can possibly meet this criterion.

Fig. 3.

Diagram of input modules (For transformable physical object, Professor Ishii pointed out a new trend of tangible median that takes a leap beyond tangible interfaces by assuming a hypothetical generation of materials that can change form and appearance dynamically [9].)

3 Design

It’s still not easy to design a framework with alterable physical input interface, as Alessandro Valli pointed out in his paper that “the problem of natural interface appearance design is closer to the creative problem of film directors and artists than it is to usability engineering [10].” Although designing a natural interface seems like a rational job, the process is pretty close to artistic creation affected by accidental factors and personal inspirations. There is no fixed route leading to successful design.

3.1 Beginning with Interaction Scheme: Enlightenment from Children’s Game

A 5 month old infant, with no rational knowledge of any object’s appearance, has already got the impulse to interact. It’s difficult and full of accidental factors to begin the design with interface appearance, and therefore the design issue may be solved from another staring point: the interaction scheme. Children’s interaction scheme is an optimal study object. Where do children learn their behaviors? Besides some training of basic actions, a lot of their behaviors are usually learned via games, traditional ones instead of electronic ones.

This fact provided us with the most critical point of breakthrough: TCG (Traditional Children’s Games). There are many benefits to start from TCGs: (1) Since children are normally weak in comprehension, TCGs are usually easy to understand and memorize; (2) There is a huge pool of TCGs to study. In the category “list of traditional children’s games” of Wikipedia, there are 79 different games, which is obviously an incomplete collection with even more to be discovered. (3) The TCGs coming down the ages are proven to be effective and popular. In fact, some games have histories of thousands of years. (4) Thanks to the fact that certain TCGs are still popular regionally today, the learning cost is even lower because learning is unnecessary for some users.

The following two elements were selected and utilized: children’s interaction scheme and physical toys in TCGs. Children’s interaction scheme is especially interesting and useful in this case.

3.2 From the Initial Idea to the Final Concept

A number of TCGs have great potential to be redesigned. To narrow down the references, TCGs using no external media are neglected, such as finger-guessing game, clapping game and hide-and-seek game. (It doesn’t mean that they are lower in value. They are removed from the list simply because they are not closely related to the topic of this paper.) There are many different TCGs using external media, and the popular ones can be categorized into 3 major types: card games, ball games and string games.

Among these three major types of TCGs, string games seemed to fit our needs better, because the output included aural elements while many musical instruments such as zither, guitar and violin have string parts. Among all those string games, the one named string figure stood out. The regulations of the string figure game are as follow: One player uses his/her fingers to create a string loop, and the other player takes it over with his/her own hands and changes it into a new loop. The game continues this way until someone fails to hold the string loop in his/her hands (Fig. 4).

Fig. 4.

String figure (Fig. 6 comes from Baidu Encyclopedia, http://baike.baidu.com/.)

The behavioral regulations and physical interfaces of the string figure game were abstracted and developed into the initial concept: MRB (Melodic Rope Box). MRB was a 30 cm x 30 cm red box with a 4 × 4 matrix of round holes at top. A user might insert several sticks into the holes and wrap strings around these sticks so as to create his/her own pattern, as shown in Fig. 5. This personalized interface might be used to create music. A user may pluck, press and rub different sections of the string to play music. (A section is a string segment between two adjacent sticks.) Different from the fixed interfaces of traditional musical instruments, MRB is an alterable and programmable interface. Users might customize this interface to suit their own preferences, and play sound of different instruments, for example stringed instruments, wind instruments, percussion instruments and keyboard instruments.

Fig. 5.

Melodic Rope Box

Continuing from MRB, in order to make it more suitable for the generation of visual elements, the red box was replaced with white table so that projection was visible on the platform. The manipulation space expanded from 30 cm x 30 cm to 60 cm x 60 cm, offering more space and freedom to users and making visual effects more impressive. In previous design, it was noticed that users had difficulty warping strings around sticks and a string tend to be curvy all along its length if the strings are thicker ones. In the final design, strings were cut into predefined various lengths, and cap them with copper terminals, which might be easily hooked onto metal sticks. This modification provided more convenience to users and making the resulting shapes tidier and more modern with its straight segments (Fig. 6). When users “played” with the strings of the interfaces designed by themselves, the projected visual elements appeared exactly where the fingers touched the strings and changed together with the melodies. Similarly, users might also interact with visual elements using the interface, and generate corresponding aural effects simultaneously.

Fig. 6.

Final design (In this case, there are altogether 12 holes along the edges of a rectangle. However, the number of holes in the platform can be customized.)

4 Input System

The feedback signals from this interface to the computer include: (1) Position information of a user’s finger in that 60 cm x 60 cm area, working like a mouse cursor for visual art creation. (2) Triggering signals from the strings, categorized as plucking, pressing and rubbing (three commonest action taken when playing a musical instrument) (3) Index number of the strings so as to identify which of them is triggered.

4.1 Blob Tracking

It’s a typical blob tracking task to obtain the finger position. There is much research and many solutions in this field, among which POT (Passive Optical Tracking) was finally chosen in this case. Compared with other solutions, POT has the advantages of low cost, high precision, reliable stability and convenience.

A POT system usually contains a projector, an infrared camera (i.e. a camera with infrared-pass filter), infrared lights and targets to track (i.e. objects with strong infrared reflectivity). In this case, the track target, namely the user’s fingers, was wrapped in infrared reflective tapes. On the software side, CCV² was chosen because it not only supports keystone correction, lens calibration, multiple cameras and cross-platform usage, but also transfers tracking information, including the target’s position, speed, acceleration and size, via TUIO protocol which is compatible with most platforms for generative art creation.

4.2 Input Detection

Unlike blob tracking, there is no standard solution for acquiring the triggering signals from the strings. The basic idea was that the physical signals of string triggering shall be converted to digital signals so that they may be processed by computers. SCMs (Single Chip Microcomputer) and sensors might be used to accomplish this task. After many trials, a largely reliable solution was found, which used conductive rubber tubes and Arduino based SCM. Conductive rubber tubes not only were used as strings in the interface, but also helped to convert physical signals to digital signals. This was because conductive rubber tubes were elastic and had a certain amount of electric resistance which would increase if they were stretched. When a user used his/her finger to touch a tube, its length changed together with its electric resistance value. In this way, the conversion from physical world to the digital world was done.

Detailed description of the scheme is as follows:

• Black conductive rubber tubes made of silicon and carbon were used.

• Outer diameter of the tubes equals 5 mm, and inner diameter equals 2 mm

• For tubes, each 10 cm has resistance of 1.5 KΩ, and their stretching after/before ratio is approximately 1.25.

• Arduino Uno was chosen as the SCM.

12 holes³ were numbered sequentially from 0 to 11, and connected with different analog input pins⁴ of Arduino. Arduino can apply a HIGH (refers to 5 V) or LOW (refers to 0 V) voltage to each hole, and might also perform AnalogRead function for each hole via analog input pins. When metal sticks, connected by strings, were inserted into these holes, each string was then connected into the circuit.

• Step 1: Confirm which pins were connected.⁵ Perform the following operations one by one on all 12 pins: set all analog input pins to input mode, and set one of them to HIGH voltage while keep the other 11 pins to LOW voltage. Then read from the pin set to HIGH voltage. If the returned value was HIGH, the pin was not connected; otherwise, the pin was connected in the circuit. All the index number of the connected pins are put into an array named A.

• Step 2: Confirm the sequence of the how pins contained in array A were connected. (1) Find the first and last pins in the sequence. Perform the following operations on all pins belong to array A: pick two of them as a combination. Set one of them (output mode) to HIGH voltage, the other to LOW voltage. All the other pins were set to input mode, and their voltage values were read and recorded. After every possible combination is implemented, all the recorded values are compared to get the maximum one. The combination which generates the maximum value is the correct one. That is to say, either one of the combination is the first pin while the other is the last. (2) Resort array A. Set the first pin (output mode) to HIGH voltage, the last (output mode) to LOW voltage. All the other pins were set to input mode, and sorted (from high to low) by the values read from themselves. The resorting is done.

• Step 3: Dynamically detect changes in electric resistance. Perform the following operations for the resorted array A: for 3 consecutive pins, connected by two strings, of array A, give the 1^st pin (output mode) HIGH voltage and 3^rd pin (output mode) LOW voltage. Read and record the change of voltage from the 2^nd pin (input mode). The increasing value from the 2^nd pin indicates the string between 2^nd and 3^rd pins is triggered while the decreasing value suggests the string between 1^st and 2^nd pins is triggered. A rapid bounce of the value implies a quick trigger (pluck) and a gentle bounce intimates a slower trigger (press/rub). Combining the data from blob tracking module (speed of the target/finger), it’s easy to distinguish between “press” and “rub”.

In this way, it is possible to convert 3 different inputs (pluck, press, rub) together with the index of the triggered string as digital signals and sent them to the development platform via Serial port. There are several issues worth noticing:

• Conductive rubber tubes were not ideal electric resistors. They had some bizarre properties that might introduce strong noise to the feedback signals. It’s important to maximize the variable range of a rubber tube’s resistance value, and use filtering algorithms to smooth out the noise. In this case, media filter⁶ was used and basically eliminated the negative influence of noise.

• A standard Arduino didn’t have enough analog input pins. (Arduino Uno was used in this case.) But if a more complicated StringWeaver with more holes and sticks is expected, analog input pins must be extended with analog multiplexer and shift register. In this case, 74HCT4051 NXP semiconductors and 74HC595 NXP semiconductors was used. The first one expanded the number of analog input pins by 8 at the cost of 3 digital output pins and 1 analog input pins, while the second expanded the number of digital output pins by 8 at the cost of 3 digital output pins. (Please pay attention that analog multiplexers have internal resistance,⁷ which would have influence on the value that analog input pin gets.)

4.3 Actuators

In order to manifest the pictures and make sound, a projector which projects image directly onto the physical input interface and a loudspeaker that generates sounds are used as actuators of StringWeaver.

On account of the popular protocol that input system adopts, it’s simple to modify the source code to adapt the framework of StringWeaver. Thanks to the rich interaction scheme that input system offers, it’s reasonable to believe StringWeaver can achieve better outcomes and give audience more pleasure as well.

Three prototype testaments are done as examples of StringWeaver:

• Case1: project Silking, modified from project Painting Rectangles ⁸ which was an exercise developed by Jeff Hendrickson using Processing. Painting Rectangles was a straightforward program that painted rectangles in random size and color following paths that affected by random factors. The original outcome (Fig. 7) looked like “worms” has a lot of ups and downs along the path. For project Silking, minor modifications were required to make it fit the StringWeaver framework. Input data were introduced instead of random parameters while sound module was added. The audience could play with the strings after weaving them in the input interface. The rectangles were painted along the strings that were triggered by audience fingers. Since the arc algorithm was kept, lines arced over the strings as the audience pulled strings at different parts. And the string interface could be changed dynamically without destroying the current visual creation during playing if the audiences want. The outcome (Fig. 7) was satisfied for its rhythm shape. Sound module was developed with Sonia (a library of Processing). Fundamental elements of music (do-re-mi-fa-sol-la-ti-do) were mapped on every string according to its position in the interface, as the scale of sound was determined by the index of sticks connected with strings. Some sort of melody was produced along with the painting process.

  Fig. 7.

  

  Project Painting Rectangles (left) and Project Silking (right)

• Case2: project Naughty Ball (Fig. 8) was a new version of bouncing ball. Bouncing ball was a common and regular program that could generate balls (physical property) through keyboard and mouse. With the help of StringWeaver, much more fun was brought into this simple program. Faster and smaller balls were shot by pulling the strings as slower bigger ones were created by pressing. Meanwhile, the bouncing boundary is defined by these strings which could be reorganized. Sound module was developed with Sonia by mapping different musical instrument tones to sticks. When the strings were touched by audiences or hit with balls, drum, piano, trumpet sound effects come up.

  Fig. 8.

  

  Project Naughty Ball (left) and Project Palette (right)

• Case3: project Palette (Fig. 8) was another interesting simple program because of StringWeaver. Colorful dots were sprayed with sound effects by striking the strings.

5 Conclusion

In conclusion, we designed a system named StringWeaver that could generate visual and audible art simultaneously, via a physical interface inspired by traditional children’s game named String Figure.

Our study explored the following new possibilities of generative art:

• Input modules could be designed basing on the interaction schemes of TCGS. In this case, the game String Figure served as the foundation of design. Finally a framework of physical interfaces suitable for generative art creation was conceived and three prototypes were made under the framework.

• In addition to the well-known advantages of PI, such as rapid and fluid manipulation, natural operation and less learning pressure, our alterable interface offers more variation and fun to the audience.

• The input and output ends of our interactive system are integrated as one. Their sharing the same space guarantees that the feedback is instant, direct and consistent. This mechanism gives improvisational performances more inspiration and makes it much easier.

• Our output contains motion pictures and sound, satisfying audience both visually and aurally. This cross modal outcome makes this piece of artwork an entertainment as well.

Future work: (1) Due to technological limits, all the strings are supposed to be connected end to end, which needs to be improved in the future. (2) In addition to the existing three input actions (i.e. plucking, pressing and rubbing), more need to be developed. (3) Signal noise may lead to misunderstanding of the actions, which means better filtering algorithms need to be developed. (4) It is possible to integrate StringWeaver into existing generative artworks so as to make them more friendly and interesting.