Mobile platform for affective context-aware systems

Highlights

• In our work, we focus on detection of affective states, their proper identification and interpretation with use of wearable and mobile devices.

• Furthermore, we formulate a method for personalization of emotion detection.

• This solution offers a non-intrusive measurement thanks to the use of wearable devices, such as wristbands.

Abstract

In our work, we focus on detection of affective states, their proper identification and interpretation with use of wearable and mobile devices. We propose a data acquisition layer based on wearable devices able to gather physiological data, and we integrate it with mobile context-aware framework. Furthermore, we formulate a method for personalization of emotion detection. This solution offers a non-intrusive measurement thanks to the use of wearable devices, such as wristbands. As means of validation of our concepts we describe a series of experiments that we conducted.

Introduction

Cognitive assistants [1] (oftentimes called “smart advisors”) for many years have been considered as one of the ultimate computer information technologies to help people in their daily activities. Today, number of such systems exist, and they are well integrated into both online services (e.g. Amazon Alexa) and mobile devices (Apple Siri) [2]. In its technology hype cycle, Gartner¹ still identifies them to be a technology with lots of potential. In order to provide constant and robust decision support, which ranges from general problems to narrow domain specific tasks, they need to process a large volume of data of great variety [3]. Moreover, they need to be able to adapt to the needs and habits of individual users in order to deliver personalized assistance. From technological point of view, they are often developed using the context-aware systems paradigm [4].

Context-aware systems (CAS) are an important class of intelligent systems that gained huge popularity over the last decade [5]. Context in such systems can be described as any information that can be used to characterize the situation of an entity [6]. This broad definition allows us to consider a variety of factors as the context: (a) physical, collected using device’s sensors, e.g. ambient light luminance and user location, (b) environmental, obtained via software services, e.g. weather and road traffic, (c) organizational, stored in electronic device, e.g. messages and events in a calendar. These values can be treated as a low-level context. Based on them, a high-level context can be generated, by which some semantics is introduced, certain interpretation occurs and some activities are recognized.

Rapid evolution of personal mobile devices such as smartphones, tablets, smartwatches and other types of wearables, forced researchers and developers to work on efficient methods for modeling and processing contextual information. In our recent research in the area of mobile context-aware systems we identified four challenges that should be met by every context-aware system [7]. These are: intelligibility, efficiency, privacy and robustness. Assuring these requirements is a major challenge for systems that operate in dynamic environment, where contextual information is constantly delivered in a streaming manner. To address the four challenges, in our previous work we proposed a human-readable rule-language that is capable of modeling and processing uncertain knowledge with an efficient rule-engine under the soft real-time constraints. Moreover, knowledge discovery methods from uncertain streaming data were proposed [7].

An important dimension of context can be related to the emotional condition of the user of a cognitive assistant [8]. Affective Computing (AfC) [9] is an interdisciplinary field of study, aimed at computer models and methods for recognizing and expressing emotions. Our main interest is in the detection and interpretation of affective states. Two aspects need to be considered: modes of data collection and ways of interpreting them in correlation with affective states that correspond to emotions. Today, most often harvested and processed information regard speech (prosody), body gestures and poses (e.g. 3D mapping, motion capture), facial expressions (e.g. visual analysis and electromyography) and physiological monitoring (e.g. blood pressure, pulse, galvanic skin response (GSR/EDA)).

In our prior work [10] we assert that a special case of a high-level context may be the emotional state of the user. In such case, number of problems to be solved appears. These include: (a) acquisition of physiological data to characterize the emotional condition of a person, in a mostly non intrusive manner, (b) integration of this data into a larger context-processing system, (c) personalization of emotion detection and identification (as there are important individual differences), and also (d) description of a high-level emotional state from low-level contextual data. This paper builds on our previous work and answers some of these challenges.

The main contribution of this paper consists in (1) proposal of data acquisition layer based on wearable devices able to gather physiological data, (2) integration of this layer with mobile context-aware framework, (3) formulation of a method for personalization of emotion detection. The rest of the paper is organized as follows: In Section 2 we discuss the main aspects of developing context-aware systems on mobile devices. Section 3 introduces the affective computing paradigm, emphasizing focus of our research. Our motivation for the development of the AfCAI platform is then discussed in Section 4. The development of the platform is described in Section 5. The evaluation of our work at its current stage is provided in Section 6. Related works are discussed in Section 7. Summary and future works are given in Section 8.