Skip to main content
SpringerLink
Log in

Using unlabeled data to improve classification of emotional states in human computer interaction

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

The individual nature of physiological measurements of human affective states makes it very difficult to transfer statistical classifiers from one subject to another. In this work, we propose an approach to incorporate unlabeled data into a supervised classifier training in order to conduct an emotion classification. The key idea of the method is to conduct a density estimation of all available data (labeled and unlabeled) to create a new encoding of the problem. Based on this a supervised classifier is constructed. Further, numerical evaluations on the EmoRec II corpus are given, examining to what extent additional data can improve classification and which parameters of the density estimation are optimal.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Baluja S (1999) Probabilistic modeling for face orientation discrimination: learning from labeled and unlabeled data. In: Proceedings of the 1998 conference on advances in neural information processing systems II (NIPS), pp 854–860

  2. Baumert JH, Frey AW, Adt M (1995) Analysis of heart rate variability. Background, method, and possible use in anesthesia. Der Anaesth 44(10):677–686

    Article  Google Scholar 

  3. Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: Proceedings conference on computational learning theory, pp 92–100

  4. Boiten FA, Frijda NH, Wientjes CJ (1994) Emotions and respiratory patterns: review and critical analysis. Int J Psychophysiol 17(2):103–128

    Article  Google Scholar 

  5. Breiman L (1996) Bagging predictors. Mach Learn 24:123–140

    MATH  MathSciNet  Google Scholar 

  6. Bruns T, Praun N (2002) Biofeedback: Ein Handbuch für die therapeutische Praxis. Vandenhoeck & Ruprecht, Germany

  7. Burkhardt F, Paeschke A, Rolfes M, Sendlmeier W, Weiss B (2005) A database of German emotional speech. In: Proceeding of interspeech, pp 1517–1520

  8. Christensen H, Fuglsang-Frederiksen A (1986) Power spectrum and turns analysis of EMG at different voluntary efforts in normal subjects. Electroencephalogr Clin Neurophysiol 64(6):528–535

    Article  Google Scholar 

  9. Cohn DA, Ghahramani Z, Jordan MI (1996) Active learning with statistical models. J Artif Intell Res 4:129–145

    MATH  Google Scholar 

  10. Cooper DB, Freeman JH (1970) On the asymptotic improvement in the out-come of supervised learning provided by additional nonsupervised learning. IEEE Trans Comput 19(11):1055–1063

    Article  MATH  Google Scholar 

  11. Cover TM (1965) Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans Electron 14(3):326–334

    Article  MATH  Google Scholar 

  12. Deng J, Schuller B (2012) Confidence measures in speech emotion recognition based on semi-supervised learning. In: proceeding of the INTERSPEECH 2012, pp 2226–2229

  13. Eckmann JP, Kamphorst SO, Ruelle D (1987) Recurrence plots of dynamical systems. Europhys Lett 4(9):973

    Article  Google Scholar 

  14. Ekman P, Friesen WV (1977) Facial action coding system. Consulting Psychologists Press, USA

    Google Scholar 

  15. Esparza J, Scherer S, Schwenker F (2012) Studying self- and active-training methods for multi-feature set emotion recognition. In: Proceeding of the partially supervised learning (PSL 11), LNCS. Springer, New York, pp 19–31

  16. Fahrenberg J, Foerster F (1982) Covariation and consistency of activation parameters. Biol Psychol 15(3–4):151–169

    Article  Google Scholar 

  17. Haynes JDD, Rees G (2006) Decoding mental states from brain activity in humans. Nature reviews. Neuroscience 7(7):523–534

    Google Scholar 

  18. Joachims T (2006) Transductive support vector machines. In: Semi-supervised learning. The MIT Press, London, pp 105–117

  19. Kanade T, Cohn J, Tian Y (2000) Comprehensive database for facial expression analysis. In: IEEE international conference on automatic face and gesture recognition, pp 46–53

  20. Karmakar C, Khandoker A, Voss A, Palaniswami M (2011) Sensitivity of temporal heart rate variability in poincar plot to changes in parasympathetic nervous system activity. BioMed Eng Online 10:1–14

    Article  Google Scholar 

  21. Kelley JF (1983) An empirical methodology for writing user-friendly natural language computer applications. In: SIGCHI conference on human factors in computing systems, pp 193–196

  22. Kestler HA, Schwenker F, Hoher M, Palm G (1995) Adaptive class-specific partitioning as a means of initializing RBF-networks. IEEE Int Conf Syst Man Cybern 1:46–49

    Google Scholar 

  23. Kierkels JJM, Soleymani M, Pun T (2009) Queries and tags in affect-based multimedia retrieval. In: IEEE international conference on multimedia and expo, ICME ’09, pp 1436–1439

  24. Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal 12(12):2067–2083

    Google Scholar 

  25. Krikler DM (1990) The QRS complex. Ann N Y Acad Sci 601(1):24–30

    Article  Google Scholar 

  26. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207

    Article  MATH  Google Scholar 

  27. Kuppens P, Van Mechelen I, Nezlek JB, Dossche D, Timmermans T (2007) Individual differences in core affect variability and their relationship to personality and psychological adjustment. Emotion 7(2):262–274

    Article  Google Scholar 

  28. Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30(3):261–273

    Article  Google Scholar 

  29. Lisetti CL, Nasoz F (2004) Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J Appl Signal Process 2004(4):1672–1687

    Article  Google Scholar 

  30. Malik M (1996) Heart rate variability: standards of measurement, physiological interpretation, and clinical use. Circulation 93(5):1043–1065

    Article  Google Scholar 

  31. Mehrabian A (1996) Pleasure–arousal–dominance: a general framework for describing and measuring individual differences in temperament. Curr Psychol 14(4):261–292

    Article  MathSciNet  Google Scholar 

  32. Mewett D, Reynolds K, Nazeran H (1999) Recurrence plot features of ECG signals. In: Conference on engineering in medicine and biology and 1999 annual fall meeting of the Biomedical Engineering Society, vol 2, p 193

  33. Picard RW (2000) Affective computing. MIT Press, Cambridge

  34. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191

    Article  Google Scholar 

  35. Pincus SM (1991) Approximate entropy as a measure of system complexity. Natl Acad Sci USA 88(6):2297–2301

    Article  MATH  MathSciNet  Google Scholar 

  36. Qiu G (2002) Indexing chromatic and achromatic patterns for content-based colour image retrieval. Pattern Recogn 35(8):1675–1686

    Article  MATH  Google Scholar 

  37. Richman JS, Moorman JR (2000) Physiological time-series analysis using approximate entropy and sample entropy. Am J Physiol Heart C 278(6):H2039–H2049

    Google Scholar 

  38. Rosenberg C, Hebert M, Schneiderman H (2005) Semi-supervised self-training of object detection models. In: Workshop on application of computer vision, pp 29–36

  39. Rösner D, Frommer J, Friesen R, Haase M, Lange J, Otto M (2012) Last minute: a multimodal corpus of speech-based user-companion interactions. In: Conference on language resources and evaluation (LREC ’12), pp 2559–2566

  40. Rudnicki M, Strumiłło P (2007) A real-time adaptive wavelet transform-based QRS complex detector. In: International conference on adaptive and natural computing algorithms, pp 281–289

  41. Russell J (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161–1178

    Article  Google Scholar 

  42. Schels M, Kächele M, Schwenker F (2012) Classification of emotional states in a Woz scenario exploiting labeled and unlabeled bio-physiological data. In: Proceedings of partially supervised learning 2011, pp 138–147

  43. Scherer KR (2005) What are emotions? and how can they be measured? Soc Sci Inf 44(4):693–727

    Article  Google Scholar 

  44. Schuller B, Valstar M, Eyben F, McKeown G, Cowie R, Pantic M (2011) AVEC 2011—the first international audio visual emotion challenge. In: Proceeding of the affective computing and intelligent, interaction, pp 415–424

  45. Schwenker F, Kestler HA, Palm G (2001) Three learning phases for radial-basis-function networks. Neural Netw 14:439–458

    Article  Google Scholar 

  46. Simson M (1981) Use of signals in the terminal QRS complex to identify patients with ventricular tachycardia after myocardial infarction. Circulation 64(2):235–242

    Article  Google Scholar 

  47. Soleymani M, Chanel G, Kierkels J, Pun T (2008) Affective characterization of movie scenes based on multimedia content analysis and user’s physiological emotional responses. In: IEEE international symposium on multimedia, pp 228–235

  48. Stemmler G (1989) The autonomic differentiation of emotions revisited: convergent and discriminant validation. Psychophysiology 26(6):617–632

    Article  Google Scholar 

  49. Stemmler G, Heldmann M, Pauls CA, Scherer T (2001) Constraints for emotion specificity in fear and anger: the context counts. Psychophysiology 38(2):275–291

    Article  Google Scholar 

  50. Strauss PM, Hoffmann H, Minker W, Neumann H, Palm G, Scherer S, Traue H, Weidenbacher U (2008) The pit corpus of German multi-party dialogues. In: International language resources and evaluation (LREC ’08), pp 2442–2445

  51. Van Boxtel A (2010) Facial EMG as a tool for inferring affective states. In: Proceedings of measuring behavior, pp 104–108

  52. Vapnik A (1998) Statistical learning theory. Wiley, New York

    MATH  Google Scholar 

  53. Vogt T, André E (2009) Exploring the benefits of discretization of acoustic features for speech emotion recognition. In: Proceeding of the interspeech, pp 328–331

  54. Walter S, Kim J, Hrabal D, Crawcour S, Kessler H, Traue H (2013) Transsituational individual-specific biopsychological classification of emotions. IEEE Trans Syst Man Cybern 43(4):988–995

    Article  Google Scholar 

  55. Wright RA, Dill JC (1993) Blood pressure responses and incentive appraisals as a function of perceived ability and objective task demand. Psychophysiology 30(2):152–160

    Article  Google Scholar 

  56. Zhang Z, Deng J, Schuller B (2013) Co-training succeeds in computational paralinguistics. In: IEEE international conference on acoustics, speech, and signal processing (ICASSP), pp 8505–8509

  57. Zhang Z, Schuller B (2012) Active learning by sparse instance tracking and classifier confidence in acoustic emotion recognition. In: Proceeding of the INTERSPEECH 2012, pp 362–365

Download references

Acknowledgments

This paper is based on work done within the Transregional Collaborative Research Center SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).

Author information

Authors and Affiliations

  1. Institute of Neural Information Processing, Ulm University, Ulm, Germany

    Martin Schels, Markus Kächele, Michael Glodek & Friedhelm Schwenker

  2. Medical Psychology, Ulm University, Ulm, Germany

    David Hrabal & Steffen Walter

Authors
  1. Martin Schels
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Markus Kächele
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Glodek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Hrabal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Steffen Walter
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Friedhelm Schwenker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Schels.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schels, M., Kächele, M., Glodek, M. et al. Using unlabeled data to improve classification of emotional states in human computer interaction. J Multimodal User Interfaces 8, 5–16 (2014). https://doi.org/10.1007/s12193-013-0133-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-013-0133-0

Keywords

Search

Navigation