Keywords

1 Introduction

Although picture books continue to be published in print-based formats, many are now offered in digital formats for reading and viewing on computers, tablets, and smartphones [1]. How picture-books are digitized varies from simply scanning print-based picture-books to make them available on digital reading devices to the creation of sophisticated software applications, or apps, that offer interactive features that alter the original format of the picture-e-book, provide new options and content for the reader, and move these digital narratives closer to the gaming or entertainment industry [2].

The main idea of this e-picture-book is originated from the old Japanese narrative paintings. In the old Japanese paintings, the story was experienced throughout a piece of horizontal paper by scrolling. This scroll of paper is called “Emaki-mono” in Japanese (Fig. 1). Emaki-mono is read by exposing an arms-length of the scroll at a time, from right to left. By scrolling and following illustrations with visual expressions indicating the next direction, the user can experience the story.

Fig. 1.
figure 1

A typical and old Japanese narrative painting and its scrolling method [3]

Full size image

In our e-picture-book “the Cloudy Lady” we have developed a system for a one-page, and none-edge picture book with 25 scenes. Each scene has illustrated images with visual guidance, like the Japanese old Emaki-mono, which indicates a certain direction. A multi-direction scrolling (similar to the scrolling method in Google) interface has been designed to enable the user to navigate freely between the stories scenes and find the story line. The aim was to determine the accuracy of user navigation to specific directions by referring to visual expressions.

Development of visual guidance of this picture book was designed according to our previous experiment results using a gaze tracking systemFootnote 1 for “Hokusai Manga”Footnote 2 which has many visual elements to navigate the reader (Fig. 2).

Fig. 2.
figure 2

Elements of visual guidance in “Hokusai Manga” and the average percentage of scrolling in the expected direction

Full size image

2 System

We developed the system using Xcode 8.1 environment, Swift 3.0.1 language and the execution device was the iPad with the operation system iOS 10.1.1.

2.1 Navigation Method

In this e-picture-book, we have 25 images (story scenes), which are arranged in five columns and five rowsFootnote 3. When opening the book, the user can see an image in the center of the display and a small piece of neighbor images on the right, left, up, bottom and corners of the display frame (Fig. 3).

Fig. 3.
figure 3

User can see an image in the center of the display and a small piece of neighbor images

Full size image

Since the multi-direction scrolling is possible in this e-picture-book, the user can scroll to move to any image by touching and dragging an image. The system automatically adjusts the intended image to the center of the display after dragging the image. The animation and sound effects start to be played once the translocation of each scene in the center part.

Each scene has been designed and illustrated to indicate a certain direction in which the author would expect the reader to follow by the expression of the illustrations, animation movements, and sound effects. We have built sphere architecture to avoid any inconvenient edge and help the user to move smoothly between the scenes (Fig. 4), Thus, a reader can follow a story in the e-picture-book with multi-direction scrolling (Fig. 5).

Fig. 4.
figure 4

Toroidal architecture to avoid any inconvenient edge and help the user to move smoothly between the scenes

Full size image
Fig. 5.
figure 5

The orange circles are the illustrations (scenes) which are dependent partially and also one sequence in the whole story line

Full size image

2.2 Illustrations with Visual Guidance

For illustrating the story of this book, we considered the composition, colors, texture, size, space, and other visual elements to make a strong visual guidance, based on the results of the experiment with the elements of “Hokusai Manga” as our reference. Figure 6 shows samples of illustrations used in our e-picture-book.

Fig. 6.
figure 6

Visual elements in each illustration (scene) (Color figure online)

Full size image

2.3 Story “The Cloudy Lady”

The story of this book is exclusively written and designed for the presented book format. It is an ethnic story with the theme of dedication and nature conservation. The story revolves around a character called the cloudy lady, who lives in the sky and has hair made of clouds. A lake is about to dry out and A group of animals that want to save the lake and asking the cloudy lady to come down and help the lake. Each scene of this book contains pictures of the lady as well as the environment and animals and also animation and sound effects which have guiding value. The cloudy lady comes to earth following the birds and with the help of a rabbit a horse and a turtle finds the drying lake and starts to rain until the lake gets full of water, she loses her cloudy hair and becomes so sad but finally with the help of butterflies she goes back to the sky.

3 Evaluation

For evaluation of the effectiveness of each image and also the guidance ability of the elements to navigate the user through the story line, we performed two types of experiment.

3.1 Experiment 1

The first experiment, was conducted with Tobii studio eye tracking camera (Tobii Pro X2-30). Six participants (three females and three males, between 20 and 30 years old) took part in this experiment. On a PC screen, each participant observed the visual information which was changing with the following order as shown in Fig. 7.

Fig. 7.
figure 7

Experiment with eye tracking camera (left) and the configuration of illustrations in the experiment using eye-tracking camera. a: blackout, b: cross mark at the center, c: object (illustration) at the center, 8 cross marks in 8 directions

Full size image

3.2 Experiment 2

For the second experiment, we asked the same six participants to use the e-picture-book application on the iPad and recorded the video while they were using the application in order to analyze their screening direction (Fig. 8).

Fig. 8.
figure 8

Recording experiment of a user exploring the picture book on the iPad

Full size image

3.3 Analysis

In the experiment with the eye-tracking camera, we used AOI (area of interest) data analyses method. In AIO method, we defined 8 areas of interest which one or two of them are aimed to be the target areas(s). We calculated the percentage of successful scrolls toward the target area(s) per each scene (Fig. 9).

Fig. 9.
figure 9

AOI (area of interest) data analyses method

Full size image

Additionally, with the gaze map pictures see Fig. 10 (left), we observed the direction of eye movements, order of the eye fixations and the fixation duration of each participant. We also made a diagram of each user’s scrolling direction by analyzing the recorded videos of the second experiment and compare them with the pre-determined storyline (Fig. 10).

Fig. 10.
figure 10

In gaze direction map each color represents the gaze direction of each participant and numbers are the order of eye movements on the picture and the circles’ size increases according to the gaze fixation duration (left). Diagram of predetermined story line (red) and the blue line is the example path traveld by a study (right) (Color figure online).

Full size image

4 Conclusion

In this project, the authors developed an e-picture book system for the iPad. The authors conducted an experiment by using images from “Hokusai Manga” to evaluate the effect of the illustrations for navigation and to determine the accuracy of user’s navigation to specific directions by referring to visual expressions. Considering the experiments result, a new e-picture book content for “the cloudy lady” was designed.

Analyzing the data of the experiments, we observed and obtained a unique result. According to the experiment with the eye tracking camera, the possibility that user navigated in the intended direction by images with strong visual expressions is so high (it could be 100% successful). However, the successful percentage in the experiment by using the e-picture book on the intended device (the iPad) is different. The resulted ratio shows that the users’ curiosity to check the neighbor scenes which are visible on the iPads’ screen effects on the scrolling direction and leads the user to go to a wrong direction. In this case the user can find her/his own original story line by connecting the various possibilities of interactions between the characters and illustrations in the picture book. Therefor the next stage of this research will be analyzing the effects of two or more images, and to understand how to make the unique reading style of this e-picture-book more attractive and enjoyable for both creator and reader. The creator will design storyline(s), and the reader can find many story lines in the book. We will continue the development and improvement of this picture book system to create a new and enjoyable experience of story narration with illustrated materials.