Keywords

1 Introduction

Ubiquitous, mobile and wearable networks unified on the internet are rapidly embedded into our daily living sphere. That means people, unfamiliar to information technology and human computer interaction issue, are becoming a large part of the users of the unified information environment. Thus we need a new concept of information environment design which does not force a person to have and use any computer skills to manage his living space in safe and stress-free. Such an information environment would provide modest and human friendly manner for users including elderly people.

Information assistance services are mostly based on social recommendation by collaborative filtering of huge number of logs of many consumers, which do not cover the differences of personal taste of each consumer. To perform personalized information assistance we need effective method for collecting decision-making data of each person with various kinds of objects without forcing prompted interactions with electronic gadgets.

This paper introduces a concept of Kansei mechanism and its modeling method through unconscious interaction with unified information environment.

2 Kansei Modeling

Subjective feature of each user’s requirement in information service can be schematically summarized as following [1];

  1. (1)

    Intuitive perception process: A user may receive some impressions viewing objects. We assume such a process as physical, physiological, psychological and cognitive levels of interpretation. In this process, a portion of the graphical features of an object, such as its colors and their combination, textures and shape, is a dominant factor in his intuitive perception. We can statistically model these relationships between some graphical features of objects and their interpretations [2, 3].

  2. (2)

    Subjective interpretation of situations: A user may show his intentional choice of assistance services according to his situation, such as time, place and occasion. Even if people physically sharing the same time, place, and occasion, each of them may feel different impression and may expect to receive different assist according to his life style and physical conditions. We assume such a process as physical, bottom-up multiple interpretation and top-down assistance-based levels of interpretation. We can model these relationships by statistical behavior log analysis.

  3. (3)

    Knowledge structure of service domain: Novice users may only have restricted knowledge on a service domain, while the others have much and well organized. Such a difference means each user is expecting his own answer according to his knowledge base in his mind. We can formalize this kind of knowledge structure as ontology. We are also relating subjective concepts on feeling with some graphical features of objects.

  4. (4)

    Feature of behavior pattern: A user often shows some specific behavior unconsciously according to his interest (or stress) on something; for instance, if he is interested in some goods, he often watches, touches and grasps them unconsciously to have a closer look at them. Thus, we can statistically analyze a degree of interest on objects by each person’s behavior log [4–6].

  5. (5)

    Tendency of decision making process: A person often makes his own decision pattern according to his view of life, which is originate from his intuitive perception process of objects and subjective interpretation process of situations, which is compared with his knowledge base in his mind, and which cause the difference in his behavior pattern in taking an action.

We can schematically summarize such relationships as shown in Fig. 1.

Fig. 1.
figure 1

Schematic model of Kansei [1]

Full size image

3 Personal Assistance Service Using Ubiquitous Environment

3.1 Kansei Modeling on Personal Preferences: Smart Sphere

Information assistance services, e.g., recommendation services, are mostly based on social recommendation by collaborative filtering of huge number of shopping logs of many consumers, which do not cover the differences of personal taste of each consumer. Otherwise a user has to register his preferring items which are referred as a template of the user’s model. Such systems force their users to answer a huge number of questionnaires to describe the individual preferences. These make users feel much stress in information assistance service.

Our basic ideas are (1) to find user’s interested and/or preferred items through observation on his behaviors in ubiquitous information environment, (2) to automatically build his preference model, and (3) to apply the model to provide suitable information service in the real world [9].

3.2 Micro, Mezzo and Macroscopic Observation

Utilizing ubiquitous sensors, we apply three observation methods to modeling each user’s preferences, which are micro, mezzo and macroscopic observation methods as shown in Fig. 2.

Fig. 2.
figure 2

Micro, mezzo and macroscopic views

Full size image
  1. (A)

    Microscopic view identifies a user by his ID tag as well as detects eye tracks by his facial image and some handling motions on some item by locally equipped cameras.

  2. (B)

    Macroscopic view covers a location of each person in a room by a matrix of global view cameras equipped on the ceiling. It also covers the overall spatial allocation and density of the people as well as the items in the room.

  3. (C)

    Mezzoscopic view extracts and traces each person’s locations and behaviors as time series data both from microscopic and macroscopic views. It also manages personal behavior log database.

3.3 Indirect Interaction in Active Observation

To enforce answering a huge number of questionnaires on users is a bottleneck in modeling personal preferences. One idea is just taking their behavior log via ubiquitous sensors without asking them, and mining some specific features by statistical analysis. Such a method is called passive observation. The problem of this method is to require long time and huge personal log to cover enough behavior data.

Our idea is to show several messages to each user, i.e., applying active observation, without expecting direct answers. If a message is informative and interesting to a user, he may pay attention, gaze, and follow the suggestion according to the message. In this process he is freely behaving by his intention without feeling any enforcement to answer to the system. In this case, monitoring each user’s behavior, i.e., responses to the messages, via ubiquitous sensors enables to attain enough behavior data effectively. This method corresponds to indirect interaction in active observation. The system can throw suitable and controlled messages to a user to build up his precise preference model without putting any stress on him. Thus, the system can statistically analyze a degree of interest on objects by each person’s behavior log effectively without a huge number of questionnaires.

4 Experimental Prototype: Smart Shop

In the business field, finding consumers’ preferences is an important issue. Point of sales systems are popularly used to detect the current consumers’ preferences as well as store management.

We have been developing an experimental prototype system, Smart Shop, as typical application of personal information assistance in shopping context [7].

  1. (1)

    Microscopic view devices: Each shelf is equipped with (a) an RFID tag reader to identify each consumer around there, (b) a facial camera to detect his face direction, (c) several item cameras to detect his behaviors related to the items, such as touching, grasping and wearing, and (d) several LCD monitors to show personal messages to him as well as to show public messages to the consumers.

  2. (2)

    Macroscopic view devices: The ceiling of the shop is equipped with camera array to cover whole area without occlusion to detect a location of each consumer at each time slice.

  3. (3)

    Smart shop servers: We have three types of database servers; which are (a) microscopic information servers to detect each behavior of the customer by image processing with his customer-ID by an RFID reader at each shelf, (b) a macroscopic server to integrate location data of consumers’ from camera array by image processing at each time slice, (c) a mezzoscopic server to integrate personal behavior log data from microscopic servers and macroscopic server and to manage the behavior log database, (d) a preference model server to statistically analyze each customer’s preferences from each of his behavior logs and to manage preference model database, and (e) a recommendation server to assist a customer in shopping.

We can expect that the consumer’s preference may appear his behavior; for instance, the order “grasp > touch > watch > ignore” shows his interest. Our assumption is that we can construct his preference model on items by behavior analysis.

Nevertheless we should note that we cannot directly estimate consumer’s preference form a single behavior itself. It should be evaluated all through his behavior log. It is because “shopping style” is roughly classified into two types; direct shopping type and survey type. For the former type, even a single touch shows a strong interest, while for the latter, a single touch is just one of them.

The system functions are as following;

  1. (1)

    If a customer comes to a shelf, its microscopic server is activated and senses his RFID to identify.

  2. (2)

    If he is interested in an item, he may stay there for a while to watch. If he is more interested in it, he may also touch and grasp it to have a closer look at it. Such a sequence can be detected by the ubiquitous cameras in the space. Thus, his personal behavior log is accumulated into the smart shop personal behavior log database.

  3. (3)

    By statistical analysis on the frequency and total elapsed time of watch, touch and grasp of each item, we can judge the shopping style and finally estimate the preference of the consumer on the items in the shop.

Through this process and iteration, the system can build up and update each customer’s preference model without forcing him to answer huge questionnaires.

5 Experiments for Kansei Modeling

5.1 Estimation of Dominant Attributes

We adopt conjoint analysis as to find the dominant attributes [8]. The sample products for the method are provided based on an orthogonal array. We analyze these products with quantification methods 1. The explanatory variables are the product attributes and the response variables are the degree of taste to the products estimated by the Smart Shop. As the result of the multiple regression analysis, the maximum |t| value of each attributes are considered as the degree of dominant attributes and the maximum degree is considered as the best dominant attribute.

5.2 Method of Recommendation Considering Dominant Attributes

The Smart Shop recommends products based on each customer’s taste when they stand in front of a digital signage device. They are recommended by order having high score as follows:

$$ Score_{p} = \mathop \sum \limits_{(a, v) \in p} \frac{{V_{a} \times S_{v} \times D_{a} }}{{S \times N_{v} }}, $$

where p, a, and v are a product, an attribute, and an attribute value. V a is a number of total attribute values the attribute a has. S v is a number of times a customer chose products which has the attribute value v. D a is a degree of dominant attributes of the attribute a. S is a total number of times a customer chose products. N v is a number of products which have attribute value v. The taste degrees of each attribute value are obtained by D a and the number of times a customer chose v. The scores of each product p are obtained by the total of the taste degree of each attribute falling under p.

We conducted an experiment to compare our implicit method by Smart Shop and previous explicit one by questionnaire. Subjects were 4 male students. The process is as follows:

  1. 1.

    A subject did shopping in Smart Shop. The products were 18 t-shirts selected randomly. The Smart Shop analyzed the subject’s action data and estimated a formula of subject’s action pattern.

  2. 2.

    The subject also did shopping in the Smart Shop. However the products were 18 t-shirts selected based on an orthogonal table. The Smart Shop estimated the taste degree of products and the degree of dominant attributes based on the action pattern formula.

  3. 3.

    The subject answered survey questions to evaluate the preference to the products with five phases (+2, +1, 0, −1, −2). We also estimated the degree of dominant attributes based on the answer.

  4. 4.

    Each recommendation products of the Smart Shop and the questionnaire data obtained by the degree of dominant attributes. Digital signage devices showed five of them based on the subject action for products. The subject evaluated the preference to the five recommendation products with five phases. This recommendation step was repeated 3 times and we got 15 product evaluations in total.

5.3 Active Observation for Quick Modeling of Dominant Attributes

In order to adopt active observation, we need stepwise estimation of preferred attributes through the temporal behavior log. In our experiment, we have simply applied rough set analysis to figure out the common attributes with column score and column index among items rather strongly interacted as “look,” “touch” and “take” actions.

According to the highest attributes at each step, the system shows explanation message through a digital signage concerning the interacting item at the moment. If some attribute is important to the specific customer, such an attribute and its value enables him to decide the value for him, which strengthen the user’s model. Otherwise the system reduces the score and index values of the attribute and its value for him.

6 Current Results and Discussions

6.1 Evaluation of Recommendation Method Considering Dominant Attributes

Table 1 shows the rates which evaluations estimated by Smart Shop have matched with user-given estimated values in questionnaire. The evaluations were divided into likes and dislikes. We separated into two cases; “like” and “dislike”. Here, the estimation value 0 in the answer to a questionnaire may differ for each user. The estimation accuracies of subject A and B were more than 61 % in both case, however, it of subject C and D were less than 50 % in case that 0 is likes.

Table 1. Estimated rate of taste by Smart Shop’s behavior observation
Full size table

Table 2 shows the degree of dominant attributes estimated by Smart Shop or with questionnaire. The dominant attribute estimated by Smart Shop were matched with another estimated with questionnaire about subject A and B. Particularly about subject A, the order of dominance were also matched with another. The decimal place’s difference about subject B suggests that Smart Shop can estimate the degree of dominant attributes more correctly than questionnaire. However, the dominant attributes of 2 methods were matched about subject C and D. These results may be caused by Smart Shop’s estimation or questionnaire.

Table 2. Dominant attribute rates of each subject (|t| value)
Full size table

Table 3 shows the ratio which subjects liked the recommended products. All subject’s results were more than 60 %. The results suggest Smart Shop with this study’s method can provide stable satisfaction to each subject.

Table 3. Good evaluation rate in the recommendation
Full size table

Recommended Products for subject A and B by Smart Shop were the same as another by questionnaire because both estimated dominant attributes were the same. On the other hand, those for subject C and D were not the same as another. Subject C rated the products recommended by Smart Shop less by 30 % than another although subject D rated them more by 6.7 %. These results indicate the need for review of the action pattern formula by Smart Shop.

Therefore we reviewed and optimized the action pattern formula of all subjects to maximize adjusted R-square. Some of the formulas have had unnecessary variables.

Table 4 shows the degree of dominant attributes estimated with optimized formulas of Smart Shop. The optimized formula of subject C made his dominant attributes, estimated by Smart Shop, agree with another by questionnaire. Those of subject A, B and D were not changed. We can consider the product recommendation by Smart Shop is able to satisfy subject C well as another by questionnaire.

Table 4. Dominant attribute rates of each subject (re-estimate)
Full size table

Finally, these results presented Smart Shop with our method attained implicit estimation of dominant attributes in 3 of 4 subjects. These also presented that, in the case of estimated dominant attributes by Smart Shop and another by questionnaire did not fit, it satisfied a subject better than modeling by questionnaire.

6.2 Evaluation of Active Observation for Quick Modeling of Dominant Attributes

In our current experiment in order to simplify the experiment process, the importance values of each item is directly assigned by the user. Step by step, the system updates estimated importance of attribute and its value and shows explanatory information on the display.

By the passive observation method, we needed randomly selected 200 items to build each user’s preference model on attribute and its value in 80 % precision. By adopting active observation, we needed randomly selected only 60 items to build the same precision model. That shows the efficiency of the active observation. If the method is adopted with the passive observation mechanism in Smart Shop, it reduces the user’s stress during behavior log acquisition and information recommendation as well.

7 Concluding Remarks

This paper proposed a concept of KANSEI modeling from the aspects of users’ needs in information service. The key issue is to computationally describe human information processing process from these aspects; (1) intuitive perception process, (2) subjective interpretation of their situations, (3) knowledge structure of service domain, (4) feature of behavior pattern, and (5) decision making process. We should notice that these aspects are in the same framework of modeling in robotics field.

As a typical example of this idea, we had shown the estimation method of product dominant attributes for each customer through behavior observation in a retail store. Also, we implemented the product recommendation system for experiments which recommends products based on each customer’s dominant attribute estimated using Smart Shop.