Skip to main content
SpringerLink
Log in

Biosignal-based user-independent recognition of emotion and personality with importance weighting

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

For modeling human intelligence, understanding emotional intelligence as well as verbal and mathematical intelligence is an important and challenging issue. In affective and personality computing, it has been reported that not only visual and audio signals but also biosignals are useful for estimating emotions and personality. Biosignals are expected to provide additional and less biased information in implicit assessment, but estimation performance can degrade if there are physiological individual differences. In this paper, considering individual physiological differences as a covariate shift, we aimed to improve the performance results in biosignal-based emotion and personality estimations. For this purpose, we constructed importance-weighted logistic regression (IW-LR) and importance-weighted support vector machine (IW-SVM), which mitigate the accuracy degradation due to physiological individual differences in the training data, and compared them with conventional LR and linear SVM (L-SVM) for estimation performance. As a result, most of the IW models outperform conventional models based on electrocardiogram (ECG) and galvanic skin response (GSR) features in emotion estimation. In the personality estimation, the IW method improves the macroaveraged F1-score for all SVM models. The best performing model (GSR model) outperformed the model with the best previously reported macroaveraged F1-score by 1.9% in personality estimation. These results indicate that importance weighting in machine learning models can reduce the effects of individual physiological differences in peripheral physiological responses and contribute to the proposal of a new model for emotion and personality estimations based on biosignals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://www.shimmersensing.com/

  2. https://pypi.org/project/PeakUtils/

  3. https://pypi.org/project/smartsvm/

References

  1. Abadi MK, Subramanian R, Kia SM, Avesani P, Patras I, Sebe N (2015) DECAF: MEG-Based multimodal database for decoding affective physiological responses. IEEE Trans Affective Comput 6(3):209–222

    Article  Google Scholar 

  2. Barral O, Jacucci G (2015) Applying physiological computing methods to study psychological, affective and motivational relevance. In: International workshop on symbiotic interaction, Springer, pp 35–46

  3. Batrinca L, Lepri B, Mana N, Pianesi F (2012) Multimodal recognition of personality traits in human-computer collaborative tasks. In: Proceedings of the 14th ACM international conference on multimodal interaction, pp 39–46

  4. Chang EJ, Rahimi A, Benini L Wu AYA (2019) Hyperdimensional computing-based multimodality emotion recognition with physiological signals. In: 2019 IEEE International conference on artificial intelligence circuits and systems (AICAS), IEEE, pp 137–141

  5. Correa JAM, Patras I (2018) A multi-task cascaded network for prediction of affect, personality, mood and social context using EEG signals. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE, pp 373–380

  6. Correa JAM, Abadi MK, Sebe N, Patras I (2018) AMIGOS: A dataset for affect, personality and mood research on individuals and groups. IEEE transactions on affective computing

  7. Costa PT, McCrae RR (1992) Revised NEO personality inventory (NEO-PI-R) and Neo five-factor inventory (NEO-FFI). Psychological Assessment Resources

  8. Cristianini N, Shawe-Taylor J et al (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press

  9. Eyben F, Wöllmer M, Valstar MF, Gunes H, Schuller B, Pantic M (2011) String-based audiovisual fusion of behavioural events for the assessment of dimensional affect. In: 9th IEEE international conference on automatic face and gesture recognition, FG 2011, IEEE Computer Society, pp 322–329

  10. Eysenck HJ (1950) Dimensions of personality, vol 5. Transaction Publishers

  11. Giżycka B, Nalepa GJ (2018) Emotion in models meets emotion in design: Building true affective games. In: 2018 IEEE games, entertainment, media conference (GEM), IEEE, pp 1–5

  12. Gjoreski M, Gjoreski H, Luštrek M, Gams M (2017) Deep affect recognition from RR intervals. In: Proceedings of the 2017 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2017 ACM international symposium on wearable computers, pp 754–762

  13. Gjoreski M, Mitrevski B, Luštrek M, Gams M (2018) An inter-domain study for arousal recognition from physiological signals. Informatica 42(1)

  14. Gross JJ, Feldman Barrett L (2011) Emotion generation and emotion regulation: One or two depends on your point of view. Emotion Review 3(1):8–16

    Article  Google Scholar 

  15. Gunes H, Pantic M (2010) Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners. In: International conference on intelligent virtual agents, Springer, pp 371–377

  16. Harper R, Southern J (2020) A Bayesian deep learning framework for end-to-end prediction of emotion from heartbeat. IEEE transactions on affective computing

  17. Hassan A, Damper R, Niranjan M (2013) On acoustic emotion recognition: compensating for covariate shift. IEEE Transactions on Audio, Speech, and Language Processing 21(7):1458–1468

    Article  Google Scholar 

  18. Huang J, Gretton A, Borgwardt K, Schölkopf B, Smola AJ (2007) Correcting sample selection bias by unlabeled data. In: Advances in neural information processing systems, pp 601–608

  19. Ivanov AV, Riccardi G, Sporka AJ, Franc J (2011) Recognition of personality traits from human spoken conversations. In: Twelfth annual conference of the international speech communication association

  20. Jung TP, Sejnowski TJ et al (2018) Multi-modal approach for affective computing. In: 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE, pp 291–294

  21. Kanamori T, Hido S, Sugiyama M (2009) A least-squares approach to direct importance estimation. J Mach Learn Res 10:1391–1445

    MathSciNet  MATH  Google Scholar 

  22. Kanamori T, Suzuki T, Sugiyama M (2012) Statistical analysis of kernel-based least-squares density-ratio estimation. Mach Learn 86(3):335–367

    Article  MathSciNet  Google Scholar 

  23. Kanamori T, Suzuki T, Sugiyama M (2013) Computational complexity of kernel-based density-ratio estimation: A condition number analysis. Mach Learn 90(3):431–460

    Article  MathSciNet  Google Scholar 

  24. Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067–2083

    Article  Google Scholar 

  25. Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2011) DEAP: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing 3 (1):18–31

    Article  Google Scholar 

  26. Kolodyazhniy V, Kreibig SD, Gross JJ, Roth WT, Wilhelm FH (2011) An affective computing approach to physiological emotion specificity: Toward subject-independent and stimulus-independent classification of film-induced emotions. Psychophysiology 48(7):908–922

    Article  Google Scholar 

  27. Komulainen E, Meskanen K, Lipsanen J, Lahti JM, Jylhä P, Melartin T, Wichers M, Isometsä E, Ekelund J (2014) The effect of personality on daily life emotional processes. PLoS One 9(10):e110907

    Article  Google Scholar 

  28. Kouw WM, Loog M (2019) A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence

  29. Landwehr N, Hall M, Frank E (2005) Logistic model trees. Mach Learn 59(1-2):161–205

    Article  Google Scholar 

  30. Lane RD, McRae K, Reiman EM, Chen K, Ahern GL, Thayer JF (2009) Neural correlates of heart rate variability during emotion. Neuroimage 44 (1):213–222

    Article  Google Scholar 

  31. Leiner D, Fahr A, Früh H (2012) EDA Positive change: A simple algorithm for electrodermal activity to measure general audience arousal during media exposure. Communication Methods and Measures 6(4):237–250

    Article  Google Scholar 

  32. Li Y, Kambara H, Koike Y, Sugiyama M (2010) Application of covariate shift adaptation techniques in brain–computer interfaces. IEEE Transactions on Biomedical Engineering 57(6):1318–1324

    Article  Google Scholar 

  33. Li Y, Zhou G, Graham D, Holtzhauer A (2016) Towards an eeg-based brain-computer interface for online robot control. Multimed Tools Appl 75(13):7999–8017

    Article  Google Scholar 

  34. Mairesse F, Walker MA, Mehl MR, Moore RK (2007) Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of Artificial Intelligence Research 30:457–500

    Article  Google Scholar 

  35. Markova V, Ganchev T (2018) Automated recognition of affect and stress evoked by audio-visual stimuli. In: 2018 Seventh balkan conference on lighting (BalkanLight), IEEE, pp 1–4

  36. McDuff D, Kaliouby R, Senechal T, Amr M, Cohn J, Picard R (2013) Affectiva-MIT facial expression dataset (AM-FED): Naturalistic and spontaneous facial expressions collected. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 881–888

  37. McKeown G, Valstar M, Cowie R, Pantic M, Schroder M (2011) The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Transactions on Affective Computing 3(1):5–17

    Article  Google Scholar 

  38. Mehta Y, Majumder N, Gelbukh A, Cambria E (2019) Recent trends in deep learning based personality detection. Artificial Intelligence Review, pp 1–27

  39. Mou W, Gunes H, Patras I (2019) Alone versus in-a-group: A multi-modal framework for automatic affect recognition. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15(2):1–23

    Article  Google Scholar 

  40. Perrin AFNM, Xu H, Kroupi E, Řeřábek M, Ebrahimi T (2015) Multimodal dataset for assessment of quality of experience in immersive multimedia. In: Proceedings of the 23rd ACM international conference on multimedia, pp 1007–1010

  41. Picard RW (2000) Affective computing. MIT Press

  42. Picard RW (2016) Automating the recognition of stress and emotion: From lab to real-world impact. IEEE Multimedia 23(3):3–7

    Article  Google Scholar 

  43. Ren Z, Qi X, Zhou G, Wang H (2014) Exploiting the data sensitivity of neurometric fidelity for optimizing eeg sensing. IEEE Internet of Things Journal 1(3):243–254

    Article  Google Scholar 

  44. Russell JA (1980) A circumplex model of affect. Journal of Personality and Social Psychology 39(6):1161

    Article  Google Scholar 

  45. Santamaria-Granados L, Munoz-Organero M, Ramirez-Gonzalez G, Abdulhay E, Arunkumar N (2018) Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS). IEEE Access 7:57–67

    Article  Google Scholar 

  46. Shimodaira H (2000) Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90(2):227–244

    Article  MathSciNet  Google Scholar 

  47. Singh N, Moneghetti KJ, Christle JW, Hadley D, Froelicher V, Plews D (2018) Heart rate variability: An old metric with new meaning in the era of using mhealth technologies for health and exercise training guidance. part two: Prognosis and training. Arrhythmia & Electrophysiology Review 7(4):247

    Article  Google Scholar 

  48. Soleymani M, Lichtenauer J, Pun T, Pantic M (2011) A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing 3(1):42–55

    Article  Google Scholar 

  49. Stemmler G, Wacker J (2010) Personality, emotion, and individual differences in physiological responses. Biological Psychology 84(3):541–551

    Article  Google Scholar 

  50. Subramanian R, Wache J, Abadi MK, Vieriu RL, Winkler S, Sebe N (2016) ASCERTAIN: Emotion And personality recognition using commercial sensors. IEEE Transactions on Affective Computing 9(2):147–160

    Article  Google Scholar 

  51. Sugiyama M (2012) Learning under non-stationarity: Covariate shift adaptation by importance weighting. In: Handbook of computational statistics, Springer, pp 927–952

  52. Sugiyama M, Suzuki T, Nakajima S, Kashima H, von Bünau P, Kawanabe M (2008) Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics 60(4):699–746

    Article  MathSciNet  Google Scholar 

  53. Terasawa N, Tanaka H, Sakti S, Nakamura S (2017) Tracking liking state in brain activity while watching multiple movies. In: Proceedings of the 19th ACM international conference on multimodal interaction, pp 321–325

  54. Tobin RM, Graziano WG, Vanman EJ, Tassinary LG (2000) Personality, emotional experience, and efforts to control emotions. Journal of Personality and Social Psychology 79(4):656

    Article  Google Scholar 

  55. Tsang IW, Kwok JT, Cheung PM, Cristianini N (2005) Core vector machines: Fast svm training on very large data sets. J Mach Learn Res 6(4)

  56. Tsuboi Y, Kashima H, Hido S, Bickel S, Sugiyama M (2009) Direct density ratio estimation for large-scale covariate shift adaptation. J Inform Process 17:138–155

    Article  Google Scholar 

  57. Tung K, Liu PK, Chuang YC, Wang SH, Wu AYA (2018) Entropy-assisted multi-modal emotion recognition framework based on physiological signals. In: 2018 IEEE-EMBS Conference on biomedical engineering and sciences (IECBES), IEEE, pp 22–26

  58. van den Burg GJ, Hero AO (2017) Fast meta-learning for adaptive hierarchical classifier design. arXiv preprint arXiv:1711.03512

  59. Vinciarelli A, Mohammadi G (2014) A survey of personality computing. IEEE Transactions on Affective Computing 5(3):273–291

    Article  Google Scholar 

  60. Wald A (1947) Foundations of a general theory of sequential decision functions. Econometrica. Journal of the Econometric Society, pp 279–313

  61. Wang SH, Li HT, Chang EJ, Wu AYA (2018) Entropy-assisted emotion recognition of valence and arousal using XGBoost classifier. In: IFIP international conference on artificial intelligence applications and innovations, Springer, pp 249–260

  62. Weisberg S (2005) Applied linear regression, vol 528. John Wiley & Sons

  63. Wilson G, Cook DJ (2020) A survey of unsupervised deep domain adaptation. ACM Trans Intell Syst Technol (TIST) 11(5):1–46

    Article  Google Scholar 

  64. Yamada M, Sugiyama M, Matsui T (2010) Semi-supervised speaker identification under covariate shift. Signal Process 90(8):2353–2361

    Article  Google Scholar 

  65. Yang HC, Lee CC (2019) An attribute-invariant variational learning for emotion recognition using physiology. In: ICASSP 2019-2019 IEEE International conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 1184–1188

  66. Zbilut JP, Thomasson N, Webber CL (2002) Recurrence quantification analysis as a tool for nonlinear exploration of nonstationary cardiac signals. Med Eng Phys 24(1):53–60

    Article  Google Scholar 

  67. Zhao S, Ding G, Han J, Gao Y (2018) Personality-aware personalized emotion recognition from physiological signals. In: IJCAI, pp 1660–1667

  68. Zohar AH, Cloninger CR, McCraty R, et al. (2013) Personality and heart rate variability: Exploring pathways from personality to cardiac coherence and health. Open Journal of Social Sciences 1(06):32

    Article  Google Scholar 

  69. Zuckerman M (1995) Good and bad humors: Biochemical bases of personality and its disorders. Psychological Science 6(6):325–332

    Article  Google Scholar 

Download references

Acknowledgements

The authors wish to thank the AMIGOS project members for sharing data. This work was partially supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI Grant Numbers 19H01120, 19H01719, and JST AIP Trilateral AI Research, Grant Number JPMJCR20G6, Japan.

Author information

Authors and Affiliations

  1. Japan Advanced Institute of Science and Technology, Nomi, Ishikawa, Japan

    Shun Katada & Shogo Okada

Authors
  1. Shun Katada
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shogo Okada
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shun Katada.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Katada, S., Okada, S. Biosignal-based user-independent recognition of emotion and personality with importance weighting. Multimed Tools Appl 81, 30219–30241 (2022). https://doi.org/10.1007/s11042-022-12711-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12711-8

Keywords

Search

Navigation