Abstract
The paper presents design and evaluation of emotional state estimator based on artificial neural networks for physiology-driven adaptive virtual reality (VR) stimulation. Real-time emotional state estimation from physiological signals enables adapting the stimulations to the emotional response of each individual. Estimation is first evaluated on artificial subjects, which are convenient during software development and testing of physiology-driven adaptive VR stimulation. Artificial subjects are implemented in the form of parameterized skin conductance and heart rate generators that respond to emotional inputs. Emotional inputs are a temporal sequence of valence/arousal annotations, which quantitatively express emotion along unpleasant-pleasant and calm-aroused axes. Preliminary evaluation of emotional state estimation is also performed with a limited set of humans. Human physiological signals are acquired during simultaneous presentation of static pictures and sounds from valence/arousal-annotated International Affective Picture System and International Affective Digitized Sounds databases.
Chapter PDF
Similar content being viewed by others
Keywords
References
Ćosić, K., Popović, S., Jovanovic, T., Kukolja, D., Slamić, M.: Physiology-Driven Adaptive VR System: Technology and Rationale for PTSD Treatment. Annual Review of CyberTherapy and Telemedicine 5, 179–191 (2007)
Popovic, S., Slamic, M., Cosic, K.: Scenario Self-Adaptation in Virtual Reality Exposure Therapy of Posttraumatic Stress Disorder. In: Roy, M.J. (ed.) Novel Approaches to the Diagnosis and Treatment of Posttraumatic Stress Disorder. NATO Security through Science Series - E: Human and Societal Dynamics, vol. 6, pp. 135–147. IOS Press, Amsterdam (2006)
Schuemie, M.J., van der Mast, C.A.P.G., Krijn, M., Emmelkamp, P.M.G.: Exploratory Design and Evaluation of a User Interface for Virtual Reality Exposure Therapy. In: Westwood, J.D., Hoffman, H.M., Robb, R.A., Stredney, D. (eds.) Medicine Meets Virtual Reality 02/10, pp. 468–474. IOS Press, Amsterdam (2002)
Gerardi, M., Rothbaum, B.O., Ressler, K., Heekin, M., Rizzo, A.: Virtual Reality Exposure Therapy Using a Virtual Iraq: Case Report. Journal of Traumatic Stress 21(2), 209–213 (2008)
Wood, D.P., Murphy, J.A., Center, K.B., Russ, C., McClay, R.N., Reeves, D., Pyne, J., Shilling, R., Hagan, J., Wiederhold, B.K.: Combat Related Post-Traumatic Stress Disorder: A Multiple Case Report using Virtual Reality Graded Exposure Therapy with Physiological Monitoring. Medicine Meets Virtual Reality 16, 556–561 (2008)
Russell, J.A.: A Circumplex Model of Affect. Journal of Personality and Social Psychology 39, 1161–1178 (1980)
Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual. Technical Report A-6. University of Florida, Gainesville, FL (2005)
Bradley, M.M., Lang, P.J.: The International Affective Digitized Sounds, 2nd edn. (IADS-2): Affective Ratings of Sounds and Instruction Manual. Technical report B-3. University of Florida, Gainesville, FL (2007)
Horvat, M., Popović, S., Bogunović, N., Ćosić, K.: Tagging Multimedia Stimuli with Ontologies. In: MIPRO (submitted, 2009)
Popović, S., Horvat, M., Ćosić, K.: Generator of Audio-Visual Stimuli for Psychotherapy and Psychological Training. In: Kozarić-Kovačić, D. (ed.) 3rd Croatian Symposium on Stress-Related Disorders, p. 40. University Hospital Dubrava, Zagreb (2008) (in Croatian)
Popović, S., Đukić, Z., Križek, V., Kukolja, D., Ćosić, K., Slamić, M.: Control of Emotional Characteristics of Stimuli in Virtual Reality Based Psychotherapy. In: Budin, L., Ribarić, S. (eds.) Proceedings of the 30th Jubilee International Convention MIPRO. CTS & CIS, vol. 3, pp. 91–96. MIPRO, Rijeka (2007)
Bradley, M.M., Lang, P.J.: Emotion and Motivation. In: Cacioppo, J.T., Tassinary, L.G., Bernston, G.G. (eds.) Handbook of Psychophysiology, 3rd edn., pp. 581–607. Cambridge University Press, Cambridge (2007)
Dawson, M.E., Schell, A.M., Filion, D.L.: The Electrodermal System. In: Cacioppo, J.T., Tassinary, L.G., Bernston, G.G. (eds.) Handbook of Psychophysiology, 3rd edn., pp. 159–181. Cambridge University Press, Cambridge (2007)
Lim, C.L., Rennie, C., Barry, R.J., Bahramali, H., Lazzaro, I., Manor, B., Gordon, E.: Decomposing Skin Conductance into Tonic and Phasic Components. International Journal of Psychophysiology 25, 97–109 (1997)
ECGSYN - A realistic ECG waveform generator, http://www.physionet.org/physiotools/ecgsyn/
McSharry, P.E., Clifford, G.D., Tarassenko, L., Smith, L.A.: A Dynamical Model for Generating Synthetic Electrocardiogram Signals. IEEE Trans. Biomed. Eng. 50(3), 289–294 (2003)
Lisetti, C., Nasoz, F.: Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals. EURASIP Journal on Applied Signal Processing 11, 1672–1687 (2004)
Wagner, J., Kim, J., André, E.: From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification. In: IEEE International Conference on Multimedia and Expo, pp. 940–943 (2005)
Rani, P., Liu, C., Sarkar, N., Vanman, E.: An Empirical Study of Machine Learning Techniques for Affect Recognition in Human-Robot Interaction. Pattern Analysis & Applications 9, 58–69 (2006)
Healey, J.: Wearable and Automotive Systems for Affect Recognition from Physiology. Technical report, Department of Computer Science, MIT, Cambridge, MA (2000)
Kim, K.H., Bang, S.W., Kim, S.R.: Emotion Recognition System using Short-Term Monitoring of Physiological Signals. Medical & Biological Engineering & Computing 42(3), 419–427 (2004)
Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion Recognition using Bio-Sensors: First Step Towards an Automatic System. In: Affective Dialogue Systems, Tutorial and Research Workshop, Kloster Irsee, Germany, June 14-16 (2004)
Leon, E., Clarke, G., Callaghan, V., Sepulveda, F.: A User-Independent Real-Time Emotion Recognition System for Software Agents in Domestic Environments. Engineering Applications of Artificial Intelligence 20(3), 337–345 (2007)
Kaya, N., Epps, H.H.: Relationship between Color and Emotion: A Study of College Students. College Student Journal 38(3), 396–405 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kukolja, D., Popović, S., Dropuljić, B., Horvat, M., Ćosić, K. (2009). Real-Time Emotional State Estimator for Adaptive Virtual Reality Stimulation. In: Schmorrow, D.D., Estabrooke, I.V., Grootjen, M. (eds) Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience. FAC 2009. Lecture Notes in Computer Science(), vol 5638. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02812-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-642-02812-0_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02811-3
Online ISBN: 978-3-642-02812-0
eBook Packages: Computer ScienceComputer Science (R0)