Mobile platform for affective context-aware systems☆
Introduction
Cognitive assistants [1] (oftentimes called “smart advisors”) for many years have been considered as one of the ultimate computer information technologies to help people in their daily activities. Today, number of such systems exist, and they are well integrated into both online services (e.g. Amazon Alexa) and mobile devices (Apple Siri) [2]. In its technology hype cycle, Gartner1 still identifies them to be a technology with lots of potential. In order to provide constant and robust decision support, which ranges from general problems to narrow domain specific tasks, they need to process a large volume of data of great variety [3]. Moreover, they need to be able to adapt to the needs and habits of individual users in order to deliver personalized assistance. From technological point of view, they are often developed using the context-aware systems paradigm [4].
Context-aware systems (CAS) are an important class of intelligent systems that gained huge popularity over the last decade [5]. Context in such systems can be described as any information that can be used to characterize the situation of an entity [6]. This broad definition allows us to consider a variety of factors as the context: (a) physical, collected using device’s sensors, e.g. ambient light luminance and user location, (b) environmental, obtained via software services, e.g. weather and road traffic, (c) organizational, stored in electronic device, e.g. messages and events in a calendar. These values can be treated as a low-level context. Based on them, a high-level context can be generated, by which some semantics is introduced, certain interpretation occurs and some activities are recognized.
Rapid evolution of personal mobile devices such as smartphones, tablets, smartwatches and other types of wearables, forced researchers and developers to work on efficient methods for modeling and processing contextual information. In our recent research in the area of mobile context-aware systems we identified four challenges that should be met by every context-aware system [7]. These are: intelligibility, efficiency, privacy and robustness. Assuring these requirements is a major challenge for systems that operate in dynamic environment, where contextual information is constantly delivered in a streaming manner. To address the four challenges, in our previous work we proposed a human-readable rule-language that is capable of modeling and processing uncertain knowledge with an efficient rule-engine under the soft real-time constraints. Moreover, knowledge discovery methods from uncertain streaming data were proposed [7].
An important dimension of context can be related to the emotional condition of the user of a cognitive assistant [8]. Affective Computing (AfC) [9] is an interdisciplinary field of study, aimed at computer models and methods for recognizing and expressing emotions. Our main interest is in the detection and interpretation of affective states. Two aspects need to be considered: modes of data collection and ways of interpreting them in correlation with affective states that correspond to emotions. Today, most often harvested and processed information regard speech (prosody), body gestures and poses (e.g. 3D mapping, motion capture), facial expressions (e.g. visual analysis and electromyography) and physiological monitoring (e.g. blood pressure, pulse, galvanic skin response (GSR/EDA)).
In our prior work [10] we assert that a special case of a high-level context may be the emotional state of the user. In such case, number of problems to be solved appears. These include: (a) acquisition of physiological data to characterize the emotional condition of a person, in a mostly non intrusive manner, (b) integration of this data into a larger context-processing system, (c) personalization of emotion detection and identification (as there are important individual differences), and also (d) description of a high-level emotional state from low-level contextual data. This paper builds on our previous work and answers some of these challenges.
The main contribution of this paper consists in (1) proposal of data acquisition layer based on wearable devices able to gather physiological data, (2) integration of this layer with mobile context-aware framework, (3) formulation of a method for personalization of emotion detection. The rest of the paper is organized as follows: In Section 2 we discuss the main aspects of developing context-aware systems on mobile devices. Section 3 introduces the affective computing paradigm, emphasizing focus of our research. Our motivation for the development of the AfCAI platform is then discussed in Section 4. The development of the platform is described in Section 5. The evaluation of our work at its current stage is provided in Section 6. Related works are discussed in Section 7. Summary and future works are given in Section 8.
Section snippets
Context-aware systems
The notion of context has been important in conceptualization of computer systems for many years. Systems that make use of such information are called context-aware systems (CAS). A general observation is that context is about evolving, structured, and shared information spaces, and that such spaces are designed to serve a particular purpose [11]. Schilit et al. [12] narrows this definition to be where you are, who you are with, and what resources are nearby. A similar definition was given in [
Models and methods in affective computing
Affective computing (AfC) is a paradigm originally proposed in 1997 by Rosalind Picard from MIT Media Lab [9]. It uses results of biomedical engineering, psychology, and artificial intelligence. It aims at allowing computer systems to detect, use, and express emotions [25]. It is a constructive and practical approach oriented mainly at improving human-like decision support as well as human–computer interaction. AfC puts interest in design and description of systems that are able to collect,
Challenges and motivation for the mobile AfC platform
Basically, we are aiming at developing a technology to detect, identify and interpret human emotional states, and then use them in operation of cognitive assistants. We believe that it can be provided based on the integration of context-aware systems and affective computing paradigms. We are planning to identify and characterize affective context data, and provide knowledge-based models to identify and interpret affects based on this data. A working name for our approach and the developed
Architecture proposal
The primary focus of our research was to extend the platform we developed for building mobile context-aware systems [5] with mechanisms for physiological context acquisition, processing and mediation. The architecture of this platform is presented in Fig. 2. It can be defined as an extension of standard Model View Controller software architectural pattern [47] that includes context and adaptability as a part of the model. The adaptable model layer of the architecture is responsible for
Experimental evaluation
AfCAI platform is implemented with the use of wearable devices. This assumes budgetary devices that can be affordable by everyone, but on the other hand these devices should have some minimal level of quality of (affective) data to ensure proper work of the platform. Here four experiments will be described that address the quality of data determination issue (Experiments 6.1–6.3) and physiological affective patterns discovery challenge (Experiment 6.4).
Related works
Our research presented in this paper lays on the edge of two separate areas: affective computing and mobile context-aware systems. As we stated before, we believe that these areas have a lot in common, and both can benefit from combining their achievements. However, related works presented here will be considered separately to make a better point of why our assumptions for combining them is desirable.
Summary and future work
In the paper we provided background on context-aware systems and their possible integration with affective computing. We introduced a software and hardware platform (AfCAI) that allows for detection, identification and interpretation of human emotional states. It uses physiological measurements provided by wearable devices and a custom context-aware framework. Ultimately, this platform could be an integral part of affective cognitive assistant technology. Our contribution includes specific
Grzegorz J. Nalepa is an engineer with degrees in computer science-artificial intelligence, and also philosophy. He has been working in the area of intelligent systems and knowledge engineering for over 15 years. He formulated the eXtended Tabular Trees rule representation method, as well as the Semantic Knowledge Engineering approach. He co-authored over 150 research papers in international journals and conferences. He coordinates GEIST—Group for Engineering of Intelligent Systems and
References (85)
- et al.
Context-aware service engineering: A survey
J. Syst. Softw.
(2009) - et al.
Uncertain context data management in dynamic mobile environments
Future Gener. Comput. Syst.
(2017) - et al.
Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty
Inform. Sci.
(2014) - et al.
Learning and inferring transportation routines
Artificial Intelligence
(2007) - et al.
Leveraging complex event processing for smart hospitals using RFID
J. Netw. Comput. Appl.
(2011) - et al.
AffectButton: A method for reliable and valid affective self-report
Int. J. Hum.-Comput. Stud.
(2013) - et al.
EMA: A process model of appraisal dynamics
Cognit. Syst. Res.
(2009) - et al.
A domain-independent framework for modeling emotion
Cogn. Syst. Res.
(2004) Autonomic nervous system activity in emotion: A review
Biol. Psychol.
(2010)- et al.
Assessing brain activations associated with emotional regulation during virtual reality mood induction procedures
Expert Syst. Appl.
(2015)
Knowledge Engineering: Building Cognitive Assistants for Evidence-based Reasoning
Can cognitive assistants disappear?
IEEE Pervasive Comput.
Toward cognitive assistants for complex decision making under uncertainty
Intell. Decis. Technol.
Providing Architectural Support for Building Context-Aware Applications
Methods for Modeling Self-Adaptive Mobile Context-Aware Sytems
An exploratory study into the use of an emotionally aware cognitive assistant
Affective Computing
AfCAI systems: Affective Computing with context awareness for ambient intelligence. Research proposal
Context is key
Commun. ACM
Context-aware computing applications
Rule-based solution for context-aware reasoning on mobile devices
Comput. Sci. Inf. Syst.
Uncertainty handling in rule-based mobile context-aware systems
Pervasive Mob. Comput.
Tracking and surveillance in wide-area spatial environments using the abstract hidden Markov model
Int. J. Pattern Recognit. Artif. Intell.
Mediators in the architecture of future information systems
Computer
Designing mediation for context-aware applications
ACM Trans. Comput.-Hum. Interact.
Resource-optimized quality-assured ambiguous context mediation framework in pervasive environments.
IEEE Trans. Mob. Comput.
A context-based mediation approach to compose semantic web services
ACM Trans. Internet Technol.
Emotion and Personality
Psychological Stress and the Coping Process
Apprasial theory
The Cognitive Structure of Emotions
Affective design patterns in computer games. Scrollrunner case study
Measures of emotion: A review
Cognit. Emotion
The nencki affective picture system (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database
Behav. Res. Methods
A circumplex model of affect
J. Personality Soc. Psychol.
The psychophysiology of emotion
An Approach to Environmental Psychology
Affective computing with primary and secondary emotions in a virtual human
Auton. Agents Multi-Agent Syst.
What are emotions? And how can they be measured?
Soc. Sci. Inf.
Toward multimodal fusion of affective cues
Cited by (42)
EMBN-MANET: A method to Eliminating Malicious Beacon Nodes in Ultra-Wideband (UWB) based Mobile Ad-Hoc Network
2023, Ad Hoc NetworksCitation Excerpt :Location-based routing [26] and location-aided routing use device location information to improve routing performance in an ad hoc environment. Ubiquitous computing and intelligent environments [27] demand location information to accommodate the participation of people from all walks of life. The physical or involuntary configuration makes every mobile node available with its position information during deployment.
Opportunistic sensing for inferring in-the-wild human contexts based on activity pattern recognition using smart computing
2020, Future Generation Computer SystemsCitation Excerpt :However, these systems require massive training data related to different phone positions as compared to position-dependent systems. Position-aware PAR systems [49–51] enable recognition of both phone position and physical activity using two or more classifier levels. Different researchers have presented various schemes to recognize physical activities using a smartphone.
Intelligence augmentation via human-AI symbiosis: Formulating wise systems for a metasociety
2023, Philosophy of Artificial Intelligence and Its Place in SocietyDriver Stress Detection from Physiological Signals by Virtual Reality Simulator
2023, Electronics (Switzerland)Emotion-based Dynamic Difficulty Adjustment in Video Games
2023, 2023 IEEE 10th International Conference on Data Science and Advanced Analytics, DSAA 2023 - ProceedingsContext Awareness in Recognition of Affective States: A Systematic Mapping of the Literature
2023, International Journal of Human-Computer Interaction
Grzegorz J. Nalepa is an engineer with degrees in computer science-artificial intelligence, and also philosophy. He has been working in the area of intelligent systems and knowledge engineering for over 15 years. He formulated the eXtended Tabular Trees rule representation method, as well as the Semantic Knowledge Engineering approach. He co-authored over 150 research papers in international journals and conferences. He coordinates GEIST—Group for Engineering of Intelligent Systems and Technologies (http://geist.re). For almost 10 years he has been co-chairing the Knowledge and Software Engineering Workshop (KESE) at KI, the German AI conference, and more recently on the Spanish CAEPIA, as well ECAI. He is the President of the Polish Artificial Intelligence Society (PSSI), member of EurAI. He is also a member of IEEE, KES, AI*IA, PTK.
- ☆
Partially funded by AGH University of Science and Technology grant and Jagiellonian University grant .