Real time emotion aware applications: A case study employing emotion evocative pictures and neuro-physiological sensing enhanced by Graphic Processor Units

Abstract

In this paper the feasibility of adopting Graphic Processor Units towards real-time emotion aware computing is investigated for boosting the time consuming computations employed in such applications. The proposed methodology was employed in analysis of encephalographic and electrodermal data gathered when participants passively viewed emotional evocative stimuli. The GPU effectiveness when processing electroencephalographic and electrodermal recordings is demonstrated by comparing the execution time of chaos/complexity analysis through nonlinear dynamics (multi-channel correlation dimension/D2) and signal processing algorithms (computation of skin conductance level/SCL) into various popular programming environments. Apart from the beneficial role of parallel programming, the adoption of special design techniques regarding memory management may further enhance the time minimization which approximates a factor of 30 in comparison with ANSI C language (single-core sequential execution). Therefore, the use of GPU parallel capabilities offers a reliable and robust solution for real-time sensing the user's affective state.

Introduction

Emotion aware computing was for a large period a neglected topic in the scientific community [1]. However, recent neuroscience findings have highlighted the critical role of emotions in a variety of cognitive functions like decision making [2], memory [3] and perception [4]. These arguments demonstrated the significance of emotional intelligence [5] not only when interacting with other people but also between human and machines [6]. Therefore, motivated research efforts investigate how to provide computers with abilities to recognize the user's emotional state and to naturally adapt to it [7]. The importance of emotion aware computing is desirable only in cases where the user should interact with the machine in order to achieve high performance during the task procedure that should be accomplished [8]. So, providing the machine with the capability to robustly sense the users’ negative feelings [9] (frustration, anger, stress, anxiety, disappointment, etc.) the appropriate feedback may be given to neutralize their mood [10] and to encourage them to improve their performance in several applications like tests controlled through computer [11], virtual gaming [12] or remote monitoring of elderly or disabled people [13], [14]. Initial research attempts have demonstrated that the core element of a successful affective computing system is its ability to emulate the ways that are employed in the communication between human beings [15]. The pioneering work of MIT group led to the introduction of the “Affective Computing” term and to the establishment of a framework that could be adopted for a successful human–computer interaction (HCI) system [16], while also dealing with the challenges that have to be faced and the expectations created by potential applications [8].

Previous research attempts have adopted communicative ways like facial expressions [17] and posture recognition [18]. However, several limitations occur since these modalities are highly dependent from the users’ personality [19] and their culture, resulting thus in enhanced inter-subject variability. Robust emotion recognition assumes the utilization of exaggerated expressions that are unlikely to be elicited in real-life situations [20]. Moreover, the use of cameras produces huge amount of data, while also communicates irrelevant information (e.g. subject's identity) which the user may be unwilling to reveal [7]. Since the aforementioned methodologies are based on the recognition of externally expressed emotions, some innermost may not be easily recognized [8]. Such feelings are not easily communicated even among human beings and may be better recognized by neuro-physiological sensing [7]. Data fusion [21] from both the central and the autonomic nervous system may create discrete emotional patterns for a wide range of emotions [22], which are poorly distinguishable otherwise. However, special care should be given to the experimental methodology used for emotion elicitation.

So, a key issue towards the achievement of a robust emotion aware computerized system is the establishment of a framework that is in close connection with the modern emotional theory assuring thus the reliable emotion elicitation. Recent trends regard emotions as behavioral attitudes related with evolutionary processes aiming to assure the human's survival and perpetuation [23], [24], [25]. Therefore, each situation may be judged as either a pleasant or an unpleasant one. Its importance modulates the activation level needed in order to confront the stimulus appeared. Erotic or life-threatening situations require higher activation degree than melancholic or relaxing occasions. Adopting this notion, a bi-directional model was proposed. According to this approach, emotional processing is governed by two motivational systems which are the appetitive approach dealing with the pleasant situations and the defensive one activated in case of life-threatening occasions. The activation of the aforementioned systems is described through the valence dimension, while the activation degree is represented by the arousal dimension. So, these affective variables form a 2D emotional space.

The International Affective Picture System (IAPS) collection adopts the aforementioned emotional model and provides a variety of affective visual stimuli as well as their normative ratings for both the arousal and valence dimension [26]. The use of this picture collection with simultaneous neurophysiological recordings demonstrated the facilitated encoding of emotional stimuli [27]. The combination of central nervous (event-related potentials/ERPs) and autonomic (electrodermal) activity revealed a significant correlation between skin conductance responses (SCRs) and the arousal ratings of the IAPS stimuli [23]. Moreover, late ERPs were more positive for emotional pictures [28], while their time course was influenced by the valence dimension [29]. A recent study investigated whether emotional processing is affected by the subject's gender. Early (N100) and mid (N200) ERPs were significantly greater for female participants during passive viewing of unpleasant pictures [30].

The bi-directional emotion model and the aforementioned neuroscience findings have not been widely adopted until now in the field of emotion aware computing. Relying on these notions, a Mahalanobis distance-based classification scheme was proposed for discriminating emotional instances selected from the IAPS collection. The output of the recognition sub-system was then used by an avatar which emulated the user's affective state by adapting its face and voice characteristics [14]. However, there was need for further improvement of the classification accuracy by applying data mining (decision trees) and pattern recognition (Support Vector Machines) techniques [31]. Towards the achievement of a reliable emotion-aware application, extended feature fusion from different neuro-physiological modalities was proposed as well as a close connection with the theoretical emotional framework and the independency of the two emotional variables. Moreover, gender specific classifiers were proposed according to [32] in order to further enhance the method's robustness which reached 81.3% for 4 emotional categories.

Despite the adequate classification accuracy that was demonstrated by these research efforts, there are several open issues that should be further investigated prior to the introduction of real world emotion aware applications. The proposed discrimination framework was developed for research purposes. It is oriented towards the achievement of the optimal result employing time-consuming computations that reduce its applicability. Moreover, it has been developed as an isolated application under controlled lab environments which may differ from generic real-life applications. So, an integrative approach should be adopted for linking the emotion methodology with the acquisition subsystem as well as with the avatar behavior-generation routines. Then, the proposed system would be able to gather short segments of neuro-physiological data which are processed within fractions of seconds. The user's affective state is recognized and serves as an input to the avatar which adapts its behavior either to mirror or to neutralize the user's affective state.

The current study investigates the feasibility of the Graphics Processing Unit (GPU) for the fast processing of neuro-physiological data. Short segments from both the central (ERPs) and the autonomic (SCRs) nervous system serve as an input to the system. These data are parallel processed during the feature extraction stage by algorithmic procedures that were re-designed in order to provide the optimal solution regarding the memory management. So, the aim of this paper is to demonstrate that the adoption of parallel processing may be greatly beneficial for the development of real-time emotion aware applications. Therefore, it is not focused to the extensive description of the parallelization techniques adopted. Moreover, it highlights some significant issues like time consumption on data transfer between host and device that should be taken into consideration during the system design in order to further minimize the execution time. So, the work's contribution lays on the introduction of a framework for the adoption of parallel programming for real time emotion-aware applications.

So, the remainder of this paper is organized as follows. In Section 2, we briefly introduce the GPU architecture as well as with special programming techniques adopted for the proper parallelization of an algorithm. Then, a brief description of the parallelized algorithms is performed. Within Section 3 results of the algorithms’ implementation and the execution time are presented in Section 4. Finally, the discussion of this paper appears in Section 4.