Skip to main content
SpringerLink
Log in

Coupled HMM-based multimodal fusion for mood disorder detection through elicited audio–visual signals

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Mood disorders encompass a wide array of mood issues, including unipolar depression (UD) and bipolar disorder (BD). In diagnostic evaluation on the outpatients with mood disorder, a high percentage of BD patients are initially misdiagnosed as having UD. It is crucial to establish an accurate distinction between BD and UD to make a correct and early diagnosis, leading to improvements in treatment and course of illness. In this study, eliciting emotional videos are firstly used to elicit the patients’ emotions. After watching each video clips, their facial expressions and speech responses are collected when they are interviewing with a clinician. In mood disorder detection, the facial action unit (AU) profiles and speech emotion profiles (EPs) are obtained, respectively, by using the support vector machines (SVMs) which are built via facial features and speech features adapted from two selected databases using a denoising autoencoder-based method. Finally, a Coupled Hidden Markov Model (CHMM)-based fusion method is proposed to characterize the temporal information. The CHMM is modified to fuse the AUs and the EPs with respect to six emotional videos. Experimental results show the promising advantage and efficacy of the CHMM-based fusion approach for mood disorder detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  • American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders (DSM-5R): American Psychiatric Pub

  • Barnes TR (1989) A rating scale for drug-induced akathisia. Br J Psychiatry 154:672–676

    Article  Google Scholar 

  • Bersani G, Polli E, Valeriani G, Zullo D, Melcore C, Capra E, Quartini A, Marino P, Minichino A, Bernabei L (2013) Facial expression in patients with bipolar disorder and schizophrenia in response to emotional stimuli: a partially shared cognitive and social deficit of the two disorders. J Neuropsychiatr Dis Treat 9:1137

    Article  Google Scholar 

  • Bou-Ghazale SE, Assaleh K (2002) A robust endpoint detection of speech for noisy environments with application to automatic speech recognition. IEEE international conference on acoustics, speech, and signal processing (ICASSP), pp IV-3808–IV-3811

  • Bozikas VP, Kosmidis MH, Tonia T, Andreou C, Focas K, Karavatos A (2007) Impaired perception of affective prosody in remitted patients with bipolar disorder. J Neuropsychiatry Clin Neurosci 19:436–440

    Article  Google Scholar 

  • Brand M, Oliver N, Pentland A (1997) Coupled hidden Markov models for complex action recognition. In: Proc. Computer Vision Pattern Recognition, pp. 201–206

  • Chen CH, Lennox B, Jacob R, Calder A, Lupson V, Bisbrown-Chippendale R, Suckling J, Bullmore E (2006) Explicit and implicit facial affect recognition in manic and depressed states of bipolar disorder: a functional magnetic resonance imaging study. Biol Psychiatry 59:31–39

    Article  Google Scholar 

  • Cheng C, Chen H, Chan Y, Su Y, Tseng C (2013) Taiwan corpora of Chinese emotions and relevant psychophysiological data—normative data for Chinese jokes. Chin J Psychol 55:555–569

    Google Scholar 

  • Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, La Torre FD (2009) Detecting depression from facial actions and vocal prosody. In: Proc. IEEE international conference on affective computing and intelligent interaction and workshops, pp 1–7

  • David DP, Soeiro-de-Souza MG, Moreno RA, Bio DS (2014) Facial emotion recognition and its correlation with executive functions in bipolar I patients and healthy controls. J Affect Disord 152:288–294

    Article  Google Scholar 

  • Ekman P (1999) Basic emotions. Handbook of cognition and emotion. Wiley, New York, pp 45–60

    Google Scholar 

  • Erguzel TT, Sayar GH, Tarhan N (2016) Artificial intelligence approach to classify unipolar and bipolar depressive disorders. Neural Comput Appl 27:1607–1616

    Article  Google Scholar 

  • Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the international conference on multimedia, 2010, pp. 1459–1462

  • Furui S (1989) Unsupervised speaker adaptation based on hierarchical spectral clustering. IEEE Trans Acoust Speech Signal Process 37(12):1923–1930

    Article  Google Scholar 

  • Giannakopoulos T (2009) A method for silence removal and segmentation of speech signals, implemented in Matlab, Department of Informatics and Telecommunications, University of Athens, Greece, Computational Intelligence Laboratory (CIL), Insititute of Informatics and Telecommunications (IIT), NCSR DEMOKRITOS, Greece

  • Glorot X, Bordes A, Bengio Y (2011) Domain adaptation for large-scale sentiment classification: a deep learning approach. In: Proceedings of the 28th international conference on machine learning (ICML-11) pp. 513–520

  • Goghari VM, Sponheim SR (2013) More pronounced deficits in facial emotion recognition for schizophrenia than bipolar disorder. Compr Psychiatry 54:388–397

    Article  Google Scholar 

  • Greco A, Valenza G, Lanata A, Rota G, Scilingo EP (2014) Electrodermal activity in bipolar patients during affective elicitation. IEEE J Biomed Health Inform 18(6):1865–1873

    Article  Google Scholar 

  • Gross JJ, Levenson RW (1995) Emotion elicitation using films. Cogn Emot 9:87–108

    Article  Google Scholar 

  • Gunes H, Pantic M (2010) Automatic, dimensional and continuous emotion recognition. Int J Synth Emot (IJSE) 1(1):68–99

    Article  Google Scholar 

  • Guy W (1976) Clinical global impression scale, The ECDEU assessment manual for psychopharmacology-revised. Volume DHEW Publ No ADM 76, vol 338, pp. 218–222

  • Hamilton M (1960) A rating scale for depression. J Neurol Neurosurg Psychiatry 23:56

    Article  Google Scholar 

  • Hermansky H, Morgan N (1994) RASTA processing of speech. IEEE Trans Speech Audio Process 2:578–589

    Article  Google Scholar 

  • Hirschfeld RM, Williams JB, Spitzer RL, Calabrese JR, Flynn L, Keck PE Jr, Lewis L, McElroy SL, Post RM, Rapport DJ (2000) Development and validation of a screening instrument for bipolar spectrum disorder: the mood disorder questionnaire. Am J Psychiatry 157:1873–1875

    Article  Google Scholar 

  • Howard N (2013) Approach towards a natural language analysis for diagnosing mood disorders and comorbid conditions. In: Proc. mexican international conference on artificial intelligence (MICAI), pp. 234–243

  • Lanata A, Greco A, Valenza G, Scilingo EP (2014) A pattern recognition approach based on electrodermal response for pathological mood identification in bipolar disorders. In: Proc. IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 3601–3605

  • Langenecker SA, Bieliauskas LA, Rapport LJ, Zubieta JK, Wilde EA, Berent S (2005) Face emotion perception and executive functioning deficits in depression. J Clin Exp Neuropsychol 27:320–333

    Article  Google Scholar 

  • Leucht S, Pitschel-Walz G, Abraham D, Kissling W (1999) Efficacy and extrapyramidal side-effects of the new antipsychotics olanzapine, quetiapine, risperidone, and sertindole compared to conventional antipsychotics and placebo. A meta-analysis of randomized controlled trials. Schizophr Res 35:51–68

    Article  Google Scholar 

  • Low LSA, Maddage NC, Lech M, Sheeber L, Allen NB (2010) Influence of acoustic low-level descriptors in the detection of clinical depression in adolescents. In: Proc. IEEE international conference on acoustics speech and signal processing (ICASSP), pp. 5154–5157

  • Low LSA, Maddage NC, Lech M, Sheeber L, Allen NB (2011) Detection of clinical depression in adolescents’ speech during family interactions. IEEE Trans Biomed Eng 58(3):574–586

    Article  Google Scholar 

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW), pp. 94–101

  • Martin O, Kotsia I, Macq B, Pitas I (2006) The eNTERFACE’05 audio-visual emotion database. In: Proceedings of the 22nd international conference on data engineering workshops (ICDEW’06), Atlanta, USA, 3–7 April 2006

  • Ooi KEB, Lech M, Allen NB (2013) Multichannel weighted speech classification system for prediction of major depression in adolescents. IEEE Trans Biomed Eng 60(2):497–506

    Article  Google Scholar 

  • Perlis RH (2005) Misdiagnosis of bipolar disorder. Am J Managed Care 11:S271–S274

    Google Scholar 

  • Rattani A, Kisku DR, Bicego M, Tistarelli M (2007) Feature level fusion of face and fingerprint biometrics. In: Proceedings of the first IEEE international conference on biometrics: theory, applications, and systems, 2007, BTAS 2007, IEEE, pp 1–6

  • Rezek L, Sykacek P, Roberts SJ (2000) Learning interaction dynamics with coupled hidden Markov models. IEE proceedings—science, measurement and technology, vol 147. no. 6

  • Saha G, Chakroborty S, Senapati S (2005) A new silence removal and endpoint detection algorithm for speech and speaker recognition applications. In: Proceedings of the 11th national conference on communications (NCC), pp. 291–295

  • Sanchez MH, Vergyri D, Ferrer L, Richey C, Garcia P, Knoth B, Jarrold W (2011) Using prosodic and spectral features in detecting depression in elderly males. In: Proc. INTERSPEECH, pp. 3001–3004

  • Schuller B, Valstar M, Eyben F, McKeown G, Cowie R, Pantic M (2011) AVEC 2011 the first international audio/visual emotion challenge. In: Proc. First int. audio/visual emotion challenge and workshop (ACII), pp. 415–424

  • Schuller B, Valstar M, Eyben F, Cowie R, Pantic M (2012) AVEC 2012—the continuous audio/visual emotion challenge. In: Proc. of int. audio/visual emotion challenge and workshop (AVEC), ACM ICMI

  • Summers M, Papadopoulou K, Bruno S, Cipolotti L, Ron MA (2006) Bipolar I and bipolar II disorder: cognition and emotion processing. Psychol Med 36:1799–1809

    Article  Google Scholar 

  • Surguladze SA, Young AW, Senior C, Brébion G, Travis MJ, Phillips ML (2004) Recognition accuracy and response bias to happy and sad facial expressions in patients with major depression. Neuropsychology 18:212

    Article  Google Scholar 

  • Tzimiropoulos G, Pantic M (2014) Gauss-newton deformable part models for face alignment in-the-wild. In: Computer vision and pattern recognition (CVPR), 2014 IEEE conference, pp. 1851–1858

  • Valstar M, Schuller B, Smith K, Eyben F, Jiang B, Bilakhia S et al (2013) AVEC 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on audio/visual emotion challenge, ACM, pp 3–10

  • Vederman AC, Weisenbach SL, Rapport LJ, Leon HM, Haase BD, Franti LM, Schallmo MP, Saunders EF, Kamali MM, Zubieta JK (2012) Modality-specific alterations in the perception of emotional stimuli in bipolar disorder compared to healthy controls and major depressive disorder. Cortex 48:1027–1034

    Article  Google Scholar 

  • Wu CH, Lin JC, Wei WL (2014) A survey on audiovisual emotion recognition: databases, features, and data fusion strategies. APSIPA transactions on signal and information processing, vol. 3, e12

  • Young R, Biggs J, Ziegler V, Meyer D (1978) A rating scale for mania: reliability, validity and sensitivity. Br J Psychiatry 133:429–435

    Article  Google Scholar 

  • Zeng Z, Pantic M, Roisman G, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence, 31.1 pp. 39–58

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Cheng-Kung University, Tainan, Taiwan

    Tsung-Hsien Yang, Chung-Hsien Wu, Kun-Yi Huang & Ming-Hsiang Su

Authors
  1. Tsung-Hsien Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chung-Hsien Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kun-Yi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming-Hsiang Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chung-Hsien Wu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, TH., Wu, CH., Huang, KY. et al. Coupled HMM-based multimodal fusion for mood disorder detection through elicited audio–visual signals. J Ambient Intell Human Comput 8, 895–906 (2017). https://doi.org/10.1007/s12652-016-0395-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-016-0395-y

Keywords

Search

Navigation