1 Introduction

Weiser introduced the concept of ubiquitous computing which portrays a vision of people and environments augmented with computational resources that provide constant and seamless information and services [1]. Now we are living in a ubiquitous world where numerous devices such as home appliances and smart devices are connected to each other. Large amount of personal data are being produced and are used as both input and output data for computers to provide personalized and adaptive services for the users. Amazon Alexa, Amazon Echo, Amazon Echo Dot, Google Home and other smart devices are some examples of smart home assistant products.

Ambient intelligence involves integrating sensors into everyday objects or devices to make them ‘smart’, which can explore their environment and allow users to intuitively interact with them to help them cope with new tasks [2]. However, there are challenges in managing sporadic personal input data from multiple devices into a single integrated system. Furthermore, it is difficult to fully understand a user’s routine by analyzing digitized data due to differing personal patterns, contexts, and unexpected variables. Therefore, our research attempts to propose an integrative framework that includes cases from multiple inputs, which can be applied in actual AI-based personal assistants that connect with other smart home appliances to design more user adaptive system.

2 Literature Review

2.1 Data in Daily Lives

Introduction of various devices, including smartphones, tablet PC and wearable devices and so on, provided researchers with an understanding of human. Among data from devices, log data has primarily been used in research to figure out statistical inferences from users [3]. Numerous researchers, such as Ørmen and Thorhauge [4], and Oliver [5] analyzed smartphones log data to understand users’ practices and norms when using smartphones. More precisely, Xu et al. learned the pattern of users’ tap events [6].

Furthermore, embedded sensors - including accelerometer, digital compass, gyroscope, GPS, microphone and camera - enabled automatic tracking and monitoring multiple dimensions of human behavior and encompassing both physical and mental status of the user [7]. Several researchers analyzed data from smartphones to understand people’s body postures and motions [8]. Also, other researchers analyzed the activity recognition of users tracking the various features of physical data. [9] Data from various sensors also led us to comprehend the mental status of the user. User’s facial expression and related emotions were driven from the image data of digital camera embedded into smart devices [10]. Some of the emotions are detectable in visual appearance such as anger, happiness, surprise and dislike while emotions such as sadness and fear are audio dominant [11]. De Silva et al. examined users’ emotion by means of combination of visual and vocal expression and suggested a hybrid approach that uses multi-modal information for emotion recognition. The digital camera of a smartphone was also used to measure heart rates in order to associate with sentimental states [12].

Recently, research of understanding human through digit data has been active under the term ‘quantified-self [13, 14].’ It means analyzing human through the self-tracking of the biological, physical, behavioral, or environmental information. Tapia et al. recognized user’s presence and activity in home environment using data from set of small sensors [15]. In addition, the terminology quantified holistic self extended the range of detected personal data by including the context and needs of individuals [16]. In this flow, attempts to understand users thoroughly including their context of daily routine have been increasing, to be the recent trend of research about using data from devices.

2.2 Case-Based Reasoning and FBS Framework

There have been numerous efforts to develop AI that can echo human intelli-gence system by mimicking human extensive web of intelligence and reasoning paradigm. to that end, case-based reasoning (CBR) solves new problems by adapting previously successful solutions to similar problems [16]. As Schank elaborates, human’s knowledge about the world is organized as memory packets holding our previous experiences that we use when facing similar problems in our life [17], CBR adapts similar method in which a case represents a packet. In CBR, the cases are stored in a case base and accessed to solve new problems. When a new problem arises, the system finds the most similar cases to offer a viable solution. To do so, CBR has a cycle of four tasks called retrieve, reuse, revise and retain in its system [18].

A case is a “contextualized piece of knowledge representing an experience that teaches a lesson fundamental to achieving the goals of the reasoner” [19]. Therefore, determining how a person’s experience is arranged as an information - the content and structure of the case - is very critical [20]. There are several methods to represent a case. Feature vector approach that represent a case as a vector of attributes paired with values. Structured vector approach which is developed around a frame-based formalism and textual approach which is represented as sets of linguistic items [21]. Hierarchical case representation offers multiple representations at different levels of abstraction possibly using multiple vocabularies [22], which consequently allows the adaptation of new approaches into the case representation such as FBS framework.

The Function Behavior Structure (FBS) framework is an extension of the FBS ontology that represents the process of designing as a set of transformations among function, behavior and structure [23, 24]. The original attempt was to rep-resent artificial objects which later extended to include process of design [25]. In the FBS framework, the Function is defined as teleology; “what the artifact is for”. The Behavior is defined as artifact’s attributes that can be derived from its structure (“what the object does”), and Structure of an object is defined as its components and their relationships (“what the object consists of”) [26]. Definition of structure can cover any physical, virtual or social artefact [27], which means that application of FBS model is not only limited to physical artefacts but also to cases which will be used for further interactions with the users.

The FBS framework has been used not only in design domains but also in building case-based reasoning systems. Table 1. shows several attempts that have applied the concept of Function Behavior and Structure into CBR. Rosenman et al. made cases of room design using FBS framework [28]. Maher et al. used the FBS framework to classify variables in design processes [29, 30]. Kim et al. used function, behavior, and structure as elements to figure out the components of weaponries [31]. In overall, we can recognize that previous research utilized FBS to describe physical outputs of design products.

Table 1. Previous researches that applied FBS framework into CBR
Full size table

3 Case Representation Framework Using FBS

This section introduces our method of designing a case by applying the concept of Function-Behavior-Structure framework. First, we arranged daily data which are generated from daily lives’ devices to make them in a manageable way to construct a case. Second, we explain the overall concept of our case according to FBS concept.

3.1 Data Concepts of Daily Lives

The ultimate goal of this research is to build a CBR based AI that recommends usage of a particular device - among a variety of devices surrounding the user- tailored to user’s specific situations. We expect the AI to be used in a ubiquitous environment where every devices including smartphone, wearable devices, home appliances and etc. are connected to the AI agent. Considering that devices people possess vary from person to person, the explicit data types and their structures cannot be rashly determined. Moreover, data that are generated by devices surrounding users exist in a complex form in terms of their usage. A single string of sensor value can be interpreted differently or several sensor values from diverse devices can be used to determine a single status of the user. For these reasons, the sensor values derived from various devices should be arranged into conceptual data forms in order to be prepared to build a case out of them. In order to manage those sporadic sensor values, our research team tried to arrange those values with the natural notion of daily lives of a person. The data generated in daily lives’ multiple devices can be classified into three concepts; (1) device usage log data, (2) quantified-self data, and (3) contextual data of the environment. Definitions of each data type and examples are provided in the following Fig. 1.

Fig. 1.
figure 1

Data concepts generated by devices in daily lives

Full size image

First, the concept of device usage log data takes account the details of devices when a user uses them. This data concept includes the power status, type of function, type of mode, the value of certain mode, timestamps of every transition, and so on. Second is the concept of user’s quantified-self data. Like the general definition of quantified-self [12, 13], our team defined this concept as data tracked by the user, that describes the user’s biological, physical, emotional and other statuses. Concept of contextual data includes every data that informs the environmental status. It covers data about environment surrounding the user. Our research team expect the data concepts - device usage log data, quantified-self data, and contextual data - to enable the management the sensor values to a rigid form of daily life, even in situations where additional devices are introduced in user’s environment.

3.2 Applying FBS Framework in Case Representation

This section introduces our method of integrating the FBS framework into the case representation. Traditional conventional case is represented through a set of features and attributes paired with a value. However, our team proposes a new method for case representation through which a case is comprised of three parts: function, behavior and structure (Fig. 2). Furthermore, we categorize our data into this framework as well, such that function represents device usage log data, behavior representing quantified-self data and structure representing contextual data.

Fig. 2.
figure 2

Case represented through feature vector representation (left), and our FBS applied method (right)

Full size image

As for the case representation, function expresses the service provided to the users. This part is linked to the device usage log data. The feature of the function represents elements, such as the status of device and information about how users control the devices. This data can deal with the features of device (such as name, size, and etc.), the timestamp when users use the device, and features of setting that can be controlled by users (such as power, mode and etc.) Second, behavior expresses the status of users through quantified-self data, that means the result of calculating raw data from sensors or devices. It does not however include the process of calculation. In our progress, quantified-self data is stored in database to save and derive the meaningful information of raw data (such as, heart beat, pedometer, and etc.) From the result, AI can recognize the status of users. For example, facial expression shows emotional features of users. Also, if the raw data is meaningful enough itself, it can come under the this part. Third, structure expresses the contextual data like the status of environment. It can be divided into two aspects. First aspect is the outdoor status around users. It includes the weather of the day, current temperature, and others. Second one is the indoor status, such as the illumination, humidity of inside space, location, and etc. This part can be used to recognize the environmental context compared with them. Therefore, through these procedures, a user’s daily life activities that is somewhat broad and abstract can be structuralized. This has the advantage of integrating data from multiple input devices.

4 Conclusion

This paper reviewed the CBR utilized in artificial intelligence, and the Function Behavior and Structure framework to integrate it into our CBR based AI. We also introduced the method of representing daily data derived from various devices into a case. For case representation, we defined three data concepts to manage the sporadic data of daily routines: device usage log data, quantified-self data, and contextual data. These data concepts are used as function, behavior, and structure features that compose a case. Through this process our team could arrange the series of sensor values into meaningful information. Moreover, we designed a represented case by integrating our own data concept and the idea of FBS framework. For future work, we will implement the actual cases and CBR system with the consideration of FBS framework. The real time demonstration and in-situ tests will take place once testbed is completed. Through this phase, we will examine our framework with the real life problems. Therefore, the research will progress into the stage of improving for more accurate and well-defined services with the consideration of user’s situation, condition, and habit through our system.