Keywords

1 Introduction

The authors have focused on the shadow, which has an inseparable relationship with the body, and researched a shadow media system that supports co-creative expressions. A major feature of this system is that by presenting shadows of varying shapes and colors (hereafter called “shadow media”), which have been artificially generated from a person’s feet, the system encourages the creation of bodily expressions through the creation of a gap between the body and the shadow [1–3]. In addition, we demonstrated that presenting the shadow of a person at a geographically separated location in the space of another allows communication with a sense of coexistence between remote locations [4]. We then performed research that utilizes the features of these types of shadow media to support the sharing of shadow media and the co-creative expressions between remote locations [5].

However, we found the following two problems in the existing system, which generates shadows by processing images of the body itself obtained using thermo cameras and other devices (Fig. 1(a)). First, because the system deals with two-dimensional images, shadow media movements are limited to two-dimensional operations. Second, it is difficult to modify the shadow media separately for individual body parts.

Fig. 1.
figure 1

Existing shadow media and shadow media using virtual 3D space

Full size image

On the other hand, in the study of Computer Graphics, there are researchers who have reproduced a three-dimensional (3D) model from the body of a person, and operated the 3D model using the data of a skeleton structure [6–9]. In addition, there are ongoing researches on real-time generation technology of a 3D shadow model created in virtual space [10].

In this study, the authors propose and develop a new shadow media system that uses a virtual 3D space and skeleton information to allow for the generation of shadow media, the design of movements for generated shadow media, and the mutual transmission of shadow media between multiple points (Fig. 1(b)). To accomplish this, this system uses a method that creates a 3D model of a person’s body that is then used to generate the shadow. By allowing the 3D model to be controlled with skeleton information, the system can control the position, pose, and movement of the shadow media in 3D. Moreover, we implement shadow media communication that uses skeleton information, which uses less data compared to image data.

By utilizing virtual 3D space and skeleton information, this system not only solves the two problems listed above but also opens the possibility of realizing shadow media robotization or shadow media expression space design as well as multi-point shadow system communication and cloud integration. In other words, this system is expected not only to support co-creative expressions, but also serve a crucial role as a platform for utilizing shadow media in a variety of fields such as rehabilitation and amusement.

In this paper, we explain the details of the system for generating shadow media using a virtual 3D space and skeleton information (hereafter called “the 3D Shadow Media System”) and introduce an actual example of shadow media generated using this system.

2 Capability of the 3D Shadow Media System

Developing the 3D Shadow Media System which consist of virtual 3D space and skeleton information could allow the researchers to realize various designs of shadow media shown in the Fig. 2.

Fig. 2.
figure 2

Capability of the 3D shadow media system

Full size image

The application of skeleton structure information will enable a local operation of the shadow media. With the skeleton structure, a local movement can be produced in each arm of the shadow (Fig. 2(a)). It also allows the shadow to flip from left to right (Fig. 2(b)). In addition, controlling the movements of the shadow media piece-by-piece allows for the generation of shadow media such as that in which the body becomes a spring (Fig. 2(c)). Then, by combining past logged data of walking movements with current skeleton information, we can separate the movement of the shadow media from the person and make it walk separately (Fig. 2(d)). In the future, it should also be possible to make the shadow media move autonomously.

The major feature of this study is that it generates shadow media not through human images, but through a combination of 3D human models and skeleton information. Thus, 3D models of other people or animals can be controlled using one’s own skeleton information. In other words, the system can replace one’s own shadow with those of other people or animals (Fig. 2(e)). In addition, because the shadow media is generated in a virtual 3D space, it can freely change the number and position of virtual light sources used with the shadow media (Fig. 2(f)) as well as display the shadows of virtual objects placed within the virtual space (Fig. 2(g)). With the introduction of a physics engine into the virtual space, it can also move shadow media in environments that differ from the actual space, such as a weightless space (Fig. 2(h)). This means that this system allows for the free design of the shadow media expression space.

Also, since this study generates shadow media from skeleton information, it implements shadow media communication by transmitting only skeleton information. Due to the fact that skeleton information requires far smaller amounts of data in comparison to image data, this allows us to increase communication speed and implement communication between multiple points, allowing shadow media from a variety of people to be shared among multiple points (Fig. 2(i)). Moreover, if we construct a shadow database that collects and stores 3D models, accessing this database would allow for the creation of arbitrary shadow media of people registered in the database from skeleton information. This means that the series of shadow media generation processes can be stored in the cloud.

3 Development of the 3D Shadow Media System

In this study, we prepared, “model data”, a 3D model of a person with skeleton information built into it in advance. In other words, by performing as much of the processing required to map skeleton information to 3D models beforehand, we ensured the real-time property of the shadow media generation process. Based on this, in this study we developed a shadow media system composed of the six construction steps given below (Fig. 3).

Fig. 3.
figure 3

Shadow media generation process using the virtual 3D space

Full size image
  1. 1.

    Create a virtual space

  2. 2.

    Obtain human skeleton information

  3. 3.

    Create model data

  4. 4.

    Operate on skeleton information (skeleton controller)

  5. 5.

    Deform model data using skeleton information

  6. 6.

    Generate and display the shadow media images

Steps 1 and 3 are performed beforehand, and steps 2 and 4 through 6 are performed in real time. The following describes the methodology of how to realize a communication of skeleton information among these processes.

  1. 1)

    Create a virtual space

Place the elements that compose the shadow media system, such as the apparatus used to measure physical images and skeleton information (such as a Kinect), the screen, and projector within the virtual space. These elements are placed in the virtual space using the same coordinates and dimensions of those in the actual space.

  1. 2)

    Obtain human skeleton information

We use Microsoft Kinect or NaturalPoint OptiTrack to obtain skeleton information. The obtained data, showing major positions of human skeleton [11] (Fig. 4(a)), is utilized to create vectors (bones) indicated in the Fig. 4(b). This has been used as skeleton information.

Fig. 4.
figure 4

Skeleton information and the direction of the vectors (bones)

Full size image
  1. 3)

    Create model data

The creation of model data consists of two steps: the creation of the 3D model of a person, and the mapping of skeleton information to the created 3D model. In this paper, we explain a method that creates model data using a Kinect.

To create the 3D model of a person, we first take depth imaging of a person in a basic posture (with arms and legs spread) from multiple directions using the Kinect. We call the skeleton information saved from that imaging “basic posture skeleton information”. Next, we generate mesh data that reproduces a 3D surface shape using the depth imaging obtained with the Kinect. By combining mesh data obtained from multiple Kinects at this point, we automatically generate a 3D model that covers all directions (Fig. 5(a)).

Fig. 5.
figure 5

Automatic generation of the 3D model using the kinect

Full size image

To map the skeleton information to the 3D model of a person, we select the bones that are closest to each vertex on the 3D model using the basic posture skeleton information obtained from the Kinect and map the data. In this manner, we generate model data in which all vertices are mapped with every bone in the skeleton information (Fig. 5(b)). By creating and storing the model data for other people using the same method, the system allows for the generation of shadow media using the model data from many people. It also allows one to generate shadow media for objects other than humans by creating models for subjects such as animals, which have different skeletons from humans, by mapping the bones.

  1. 4)

    Operate on skeleton information (Skeleton Controller)

In this study, we implement deformation of individual parts of the skeleton information through performing the following operations, either independently or in tandem, on arbitrary bones within the skeleton information.

  • Vary the time of the final positional coordinates of bones using an arbitrary function

  • Apply a transformation matrix to bone vectors for rotation or size manipulation

In addition, the ends of all bones in the current skeleton information and the skeleton information processed a frame earlier are connected to one another by springs and dampers. This system allows the skeleton to oscillate or attenuate in reaction to human body movements. (See the Fig. 6.)

Fig. 6.
figure 6

The shadow media with the oscillated or attenuated movements

Full size image
  1. 5)

    Deform model data using skeleton information

To deform the shadow media by operating on the skeleton information using the skeleton controller, we must modify the pose of the 3D model of a person. The processing method is explained below. First, using the skeleton information bones of the 3D model created in 3) and the skeleton information bones operated on by the skeleton controller, we calculate a transformation matrix for each respective bone. Next, we use a transformation matrix corresponding to the bone mapped to each vertex in model data in 3) to transform coordinates. Modifying the vertices that compose the 3D model in this fashion, the 3D model is deformed to the skeleton information pose operated on by the skeleton controller as shown in Fig. 7.

Fig. 7.
figure 7

The transformation of the model data using skeleton information

Full size image
  1. 6)

    Generate and display the shadow media image

In this system, we use virtual light sources placed at arbitrary points in the virtual space, the 3D model generated in 5), and a screen to generate shadows of 3D models on a screen in the virtual space (Fig. 8(a)). Specifically, we first create projection images of the 3D model from the positions of the virtual light sources. Next, we cut rectangles of the projection image where the projection range of the projector (a quadrangular pyramid) and screen intersect (the “projection surface”). Finally, we map the projection image cut from the projection surface to the projection surface. As a result, a black 3D model image is projected in the correct position. Obtaining that shadow image as a projection image as seen from the position of the projector allows for the generation of a shadow image as projected from the projector in actual space (Fig. 8(b)).

Fig. 8.
figure 8

Generate and project the shadow media image

Full size image

Finally, we perform media processing such as modifications to color and shape on the created shadow image. This image is called the shadow media image, and is displayed using the projector.

  1. 7)

    Communication of the skeleton information

In this study, we are aiming at cloudizing the shadow media system by dividing the above-mentioned 1 through 6 processes of shadow generation and by building connections among the processes via IP communication technology. The following is the description of its outline.

First, we have dividend the work of shadow media generation into three processes: obtaining skeleton information, operating the information and applying the information to model data to generate shadow media.

So as to realize data communications among those three processes, we have developed a server software to achieve a multi-point P2P communication via TURN (Traversal Using Relay NAT) [12] methods, an encoder that is able to define data structure freely. A series of systems used in this study have been named CIPC (Central Inter-process Communication). With data buffer prepared in the server, these systems enable a one-on-multi people communication among the processes. This feature separates CIPC from the conventional TURN methods.

The above mentioned steps have allowed us to divide the shadow media processes, achieve arbitrary P2P communications among the divided processes and have prospects of success for cloudization of the shadow media system. The Fig. 9 shows the shadow media system developed in this study.

Fig. 9.
figure 9

The structure of the new shadow media system

Full size image

4 Utilization of the 3D Shadow Media System

4.1 The Shadow Media using the Skeleton Controller

Fig. 10 shows examples of using the skeleton controller to modify parts of the shadow media individually. Fig. 10(a) shows shadow media with periodic motion (2[s] period) added to the movement of the hips. Fig. 10(b) shows shadow media with an arm deformed to bend unnaturally. In this fashion, the system allows the projection of shadow media in poses that humans could not normally achieve. In addition, Fig. 11 shows shadow media in which the person’s movements oscillate like a spring, and shadow media in which movements are attenuated, being displayed. As a result, participants who experienced shadow media being oscillated reported feeling a sensation of being pulled. At the same time, participants who experienced shadow media being attenuated reported feeling as though they were in a heavy space. These results demonstrate the possibility of using the system to control shadow media movement to evoke transformations in physical sensations.

Fig. 10.
figure 10

Every part of the shadow media modified individually

Full size image
Fig. 11.
figure 11

The shadow media moving like springs and dampers

Full size image

4.2 The Shadow Media using the Virtual 3D Space

Fig. 12 shows examples of virtual objects that do not exist within the actual space being placed in the virtual 3D space, and their shadows being displayed. In Fig. 12(a), the shadow of a 3D model mimicking the shape of a chair is projected in the actual space. In Fig. 12(b), multiple spherical objects are floated in the air of the virtual space, and their shadows are projected in the actual space. This system also allows for modification of the positions of virtual light sources for the shadow media. In Fig. 12(c), the virtual light source is positioned at eye-level front and center with respect to the screen. In contrast, in Fig. 12(d) the virtual light source is placed above the person’s head. In addition, this system also allows switching the shadow of the person for the shadows of other people or animals. In Fig. 12(e), shadow media created using female model data is displayed for a male. In Fig. 12(f), shadow media created by applying the 3D model of a cat to a person is displayed. Those results demonstrate that the utilization of the inseparability between the shadow media and human body allows various articles to be positioned in a space where a person exists while relating those items to the person. By using the inseparability, a person can be transformed into another person or being: an adult, a child or an animal. Moreover, the outcome shows that the relation between human body and shadow can be changed in a various way through spatial or temporal operation of the virtual light source.

Fig. 12.
figure 12

The shadow media using the virtual 3D space

Full size image

4.3 The Shadow Media Communication using CIPC

We have carried out a shadow media communication between remote locations by conducting the following processes; transmitting skeleton information mutually between geographically remote locations then utilizing the transmitted data to generate shadow media at each location (Fig. 13(a)).

Fig. 13.
figure 13

The shadow media communication using skeleton information

Full size image

The Fig. 13(b) shows application of the skeleton information of male person at the location A to a female model data. After the information is applied to it, a shadow media is generated and transmitted to the location B.

5 Summary

In this study, we proposed and developed a system for generating shadow media using a virtual 3D space and skeleton information. Specifically, we developed a skeleton controller capable of operating on skeleton information and constructed a system that automatically generates and deforms 3D models of people and projects shadows generated in a virtual 3D space in an actual space. As a result, the system can freely control the pose of the shadow media as well as the movements of individual parts of the shadow. Moreover, the system displays the shadows of other people or animals as one’s own shadow media, modifies the position of virtual light sources for the shadow media, and displays shadow media for objects that do not exist in the actual space. By the effects of the inseparability between the human shadow and body and of the bodily wholeness observed in the human shadow, this system enables creation of a media space of expression, different from C-Ged one, and generation of a new shadow media or shadow media expression space. In future work, we would like to utilize this system as a platform for media systems that support collaborative expression.