Skip to main content
SpringerLink
Log in

Affect detection from non-stationary physiological data using ensemble classifiers

  • Original Paper
  • Published:
Evolving Systems Aims and scope Submit manuscript

Abstract

Affect detection from physiological signals has received considerable attention. One challenge is that physiological measures exhibit considerable variations over time, making classification of future data difficult. The present study addresses this issue by providing insights on how diagnostic physiological features of affect change over time. Affective physiological data (electrocardiogram, electromyogram, skin conductivity, and respiration) was collected from four participants over five sessions each. Classification performance of a number of training strategies, under different conditions of features selection and engineering, were compared using an adaptive classifier ensemble algorithm. Analysis of the performance of individual physiological channels for affect detection is also provided. The key result is that using pooled features set for affect detection is more accurate than using day-specific features. A decision fusion strategy which combines decisions from classifiers trained on individual channels data outperformed a features fusion strategy. Results also show that the performance of the ensemble is affected by the choice of the base classifier and the alpha factor used to update the member classifiers of the ensemble. Finally, the corrugator and zygomatic facial EMGs were found to be more reliable measures for detecting the valence component of affect compared to other channels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Allanson J, Fairclough SH (2004) A research agenda for physiological computing. Interact Comput 16(5):857–878

    Article  Google Scholar 

  • Alzoubi O (2012) Automatic affect detection from physiological signals: practical issues. PhD thesis, University of Sydney, Camperdown

  • AlZoubi O, D’Mello SK, Calvo RA (2012) Detecting naturalistic expressions of nonbasic affect using physiological signals. IEEE Trans Affect Comput 3(3):298–310

    Article  Google Scholar 

  • AlZoubi O, Calvo RA, Stevens RH (2009) Classification of EEG for affect recognition: an adaptive approach. In: Nicholson A, Li X (eds) AI 2009: advances in artificial intelligence. Springer, Heidelberg, pp 52–61

  • AlZoubi O, Hussain MS, D’Mello S, Calvo RA (2011) Affective modeling from multichannel physiology: analysis of day differences. In: Proceedings of the 4th international conference on affective computing and intelligent interaction, vol I. Springer, Heidelberg, pp 4–13

  • Andreassi JL (2007) Psychophysiology: human behavior and physiological response, 5th edn. Lawrence Erlbaum Associates, New Jersey

    Google Scholar 

  • Angelov P, Filev DP, Kasabov N (2010) Evolving intelligent systems: methodology and applications, vol 12. Wiley, New York

    Book  Google Scholar 

  • Bifet A, Holmes G, Pfahringer B, Kirkby R, Gavaldà R (2009) New ensemble methods for evolving data streams. In: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining, ACM, New York, NY, USA, KDD ’09, pp 139–148

  • Bradley M, Lang PJ (2007) The international affective picture system (IAPS) in the study of emotion and attention. Oxford University Press, New York, pp 29–46

    Google Scholar 

  • van den Broek EL, Schut MH, Westerink JHDM, Tuinenbreijer K (2009) Unobtrusive sensing of emotions (use). J Ambient Intell Smart Environ 1(3):287–299

    Google Scholar 

  • Chanel G, Kronegg J, Grandjean D, Pun T (2006) Emotion assessment: arousal evaluation using eegs and peripheral physiological signals. In: Gunsel B, Jain AK, Tekalp AM, Sankur B (eds) Multimedia content representation, classification and security. Springer, Heidelberg, pp 530–537

  • Cieslak D, Chawla N (2009) A framework for monitoring classifiers performance: when and why failure occurs? Knowl Inf Syst 18(1):83–109

    Article  Google Scholar 

  • Duda R, Hart P, Stork D (2001) Pattern classification. Wiley, New York

    MATH  Google Scholar 

  • Ekman P (1992) An argument for basic emotions. Cognit Emot 6(3):169–200

    Article  Google Scholar 

  • Ekman P (1994) Moods, emotions and traits. Oxford University Press, New York, 56–58

    Google Scholar 

  • Gomez P, Zimmermann PG, Schär SG, Danuser B (2009) Valence lasts longer than arousal. J Psychophysiol 23(1):7–17

    Article  Google Scholar 

  • Haag A, Goronzy S, Schaich P, Williams J (2004) Emotion recognition using bio-sensors: first steps towards an automatic system. In: André E, Dybkjaer L, Minker W, Heisterkamp P (eds) Affective dialogue systems. Springer, Heidelberg, pp 36–48

  • Hamm AO, Schupp HT, Weike AI (2003) Motivational organization of emotions: autonomic changes, cortical responses, and reflex modulation. In: Davidson RJ, Scherer KR, Goldsmith HH (eds) Handbook of affective sciences. Oxford university press, Oxford, UK, pp 187–211

  • Heijden F, Duin R, Ridder D, Tax D (2004) Classification, parameter estimation and state estimation—an engineering approach using Matlab. Wiley, Chichester, UK

    Book  MATH  Google Scholar 

  • Jain AK, Duin RPW, Jianchang M (2000) Statistical pattern recognition: a review. IEEE Trans Pattern Anal Mach Intell 22(1):4–37

    Article  Google Scholar 

  • Kim K, Bang S, Kim S (2004) Emotion recognition system using short-term monitoring of physiological signals. Medical Biol Eng Comput 42(3):419–427

    Article  Google Scholar 

  • Kim J, André E (2006) Emotion recognition using physiological and speech signal in short-term observation. Perception and interactive technologies. Springer, Berlin, pp 53–64

    Chapter  Google Scholar 

  • Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067–2083

    Article  Google Scholar 

  • Kolter JZ, Maloof M (2003) Dynamic weighted majority: a new ensemble method for tracking concept drift. In: Third IEEE international conference on data mining, ICDM 2003, 2003. IEEE, pp 123–130

  • Kreibig SD (2010) Autonomic nervous system activity in emotion: a review. Biol Psychol 84(3):394–421

    Article  Google Scholar 

  • Kuncheva L (2004a) Classifier ensembles for changing environments. In: Roli F, Kittler J, Windeatt T (eds) Multiple classifier systems, vol 3077., Lecture notes in computer scienceSpringer, Berlin, pp 1–15

    Chapter  Google Scholar 

  • Kuncheva L (2004b) Combining pattern classifiers: methods and algorithms. Wiley, Hoboken, NJ

    Book  Google Scholar 

  • Kuncheva L, Christy T, Pierce I, Mansoor S (2011) Multi-modal biometric emotion recognition using classifier ensembles. In: Mehrotra K, Mohan C, Oh J, Varshney P, Ali M (eds) Modern approaches in applied intelligence, vol 6703., Lecture notes in computer scienceSpringer, Berlin, pp 317–326

    Chapter  Google Scholar 

  • Lang PJ, Bradley MM, Cuthbert BN et al (2005) International affective picture system (IAPS): affective ratings of pictures and instruction manual. NIMH, Center for the Study of Emotion and Attention, University of Florida, Gainesville, FL

  • Lang PJ (1995) The emotion probe. studies of motivation and attention. Am Psychol 50(5):372–385

    Article  Google Scholar 

  • Lang PJ, Bradley MM, Cuthbert BN (1995) International affective picture system (IAPS): technical manual and affective ratings. The Center for Research in Psychophysiology, University of Florida, Gainesville, FL

    Google Scholar 

  • Last M (2002) Online classification of nonstationary data streams. Intell Data Anal 6(2):129–147

    MATH  MathSciNet  Google Scholar 

  • Lazarus R (1991) Emotion and adaptation. Oxford University Press, New York

    Google Scholar 

  • Lee H, Shackman A, Jackson D, Davidson R (2009) Test–retest reliability of voluntary emotion regulation. Psychophysiol 46(4):874–879

    Article  Google Scholar 

  • Levenson RW (1992) Autonomic nervous system differences among emotions. Psychol Sci 3(1):23–27

    Article  Google Scholar 

  • Lichtenstein A, Oehme A, Kupschick S, Jrgensohn T, (2008) Comparing two emotion models for deriving affective states from physiological data. In: Peter C, Beale R (eds) Affect and emotion in human–computer interaction, vol 4868. Lecture notes in computer science. Springer, Berlin, pp 35–50

  • Lowne DR, Roberts SJ, Garnett R (2010) Sequential non-stationary dynamic classification with sparse feedback. Pattern Recognit 43(3):897–905

    Article  MATH  Google Scholar 

  • Maier-Hein L, Metze F, Schultz T, Waibel A (2005) Session independent non-audible speech recognition using surface electromyography. In: IEEE workshop on automatic speech recognition and understanding, 2005, pp 331–336

  • Muhlbaier M, Polikar R (2007) An ensemble approach for incremental learning in nonstationary environments. In: Haindl M, Kittler J, Roli F (eds) Multiple classifier systems, vol 4472. Springer, Berlin, pp 490–500

    Chapter  Google Scholar 

  • Nishida K, Yamauchi K, Omori T (2005) ACE: adaptive classifiers–ensemble system for concept-drifting environments. In: Oza N, Polikar R, Kittler J, Roli F (eds) Multiple classifier systems. Springer, Berlin. Lecture notes in computer science, vol 3541, pp 176–185

  • Oza NC, Russell S (2001) Online bagging and boosting. In: Richardson T, Jaakkola T (eds) Artificial intelligence and statistics. Morgan Kaufmann, Los Angeles, CA, pp 105–112

  • Oza NC, Tumer K (2008) Classifier ensembles: select real-world applications. Inf Fus 9(1):4–20

    Article  Google Scholar 

  • Picard RW (1997) Affective computing, 2nd edn. The MIT Press, Cambridge, MA

    Google Scholar 

  • Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191

    Article  Google Scholar 

  • Plarre K, Raij A, Hossain M, Ali A, Nakajima M, al Absi M, Ertin E, Kamarck T, Kumar S, Scott M, Siewiorek D, Smailagic A, Wittmers L (2011) Continuous inference of psychological stress from sensory measurements collected in the natural environment. In: 10th international conference on information processing in sensor networks (IPSN), 2011, pp 97–108

  • Polikar R, Upda L, Upda SS, Honavar V (2001) Learn++: an incremental learning algorithm for supervised neural networks. IEEE Trans Syst Man Cybernet C: Applicat Rev 31(4):497–508

    Article  Google Scholar 

  • Polikar R (2006) Ensemble based systems in decision making. IEEE Circuits Syst Mag 6(3):21–45

    Article  Google Scholar 

  • Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39:1161–1178

  • Russell JA, Weiss A, Mendelsohn GA (1989) Affect grid: a single-item scale of pleasure and arousal. J Personal Soc Psychol 57(3):493–502

    Article  Google Scholar 

  • Sayed-Mouchaweh M, Lughofer E (2012) Learning in non-stationary environments. Springer, New York

    Book  MATH  Google Scholar 

  • Vyzas E, Picard RW (1998) Affective pattern classification. In: Canamero D (ed) Emotional and intelligent: the tangled knot of cognition. Proceedings of the AAAI fall Symposium series. AAAI, Menlo Park, CA, pp 176–182

  • Wagner J (2009) Augsburg biosignal toolbox (aubt). http://hcm-lab.de/files/project_content/33/219_AuBTGuide.pdf. Accessed 25 Apr 2014

  • Wagner J, Kim J, André E (2005) From physiological signals to emotions: implementing and comparing selected methods for feature extraction and classification. In: IEEE international conference on multimedia and expo, ICME 2005, pp 940–943

  • Webb AR (2002) Statistical pattern recognition. Wiley, New Jersey

    Book  MATH  Google Scholar 

  • Whang M, Lim J (2008) A physiological approach to affective computing. In: Affective computing: focus on emotion expression, synthesis, and recognition. I-Tech Education and Publishing, Vienna, Austria, pp 310–318

  • Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Mach Learn 23(1):69–101

    Google Scholar 

  • Witten I, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn., Series in data management systems. Morgan Kaufmann, Burlington, MA

    Google Scholar 

  • Yue S, Guojun M, Xu L, Chunnian L (2007) Mining concept drifts from data streams based on multi-classifiers. In: 21st international conference on advanced information networking and applications workshops, AINAW ’07, 2007, vol 2, pp 257–263

  • Zeng ZH, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1):39–58

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science, Carnegie Mellon University in Qatar, Doha, Qatar

    Omar AlZoubi & Davide Fossati

  2. Department of Computer Science, The University of Notre Dame, Notre Dame, IN, USA

    Sidney D’Mello

  3. School of Electrical and Information Engineering, The University of Sydney, Sydney, NSW, Australia

    Rafael A. Calvo

Authors
  1. Omar AlZoubi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Davide Fossati
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sidney D’Mello
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rafael A. Calvo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Omar AlZoubi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

AlZoubi, O., Fossati, D., D’Mello, S. et al. Affect detection from non-stationary physiological data using ensemble classifiers. Evolving Systems 6, 79–92 (2015). https://doi.org/10.1007/s12530-014-9123-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12530-014-9123-z

Keywords

Search

Navigation