Automatic depression classification based on affective read sentences: Opportunities for text-dependent analysis

Highlights

• During read aloud tasks, speech disfluency analysis indicates that speakers with depression have increases in hesitations and speech errors when compared to a nondepressed population.

• This study shows, in a collective sense, that all ranges of affective valence speech samples (e.g. negative, neutral, positive) are important in the speech-based analysis of depression.

• By fusing multi-valence based features from specific sentence groups, significant automatic depression classification improvements were recorded when compared to the affect agnostic feature baseline.

Abstract

In the future, automatic speech-based analysis of mental health could become widely available to help augment conventional healthcare evaluation methods. For speech-based patient evaluations of this kind, protocol design is a key consideration. Read speech provides an advantage over other verbal modes (e.g. automatic, spontaneous) by providing a clinically stable and repeatable protocol. Further, text-dependent speech helps to reduce phonetic variability and delivers controllable linguistic/affective stimuli, therefore allowing more precise analysis of recorded stimuli deviations. The purpose of this study is to investigate speech disfluency behaviors in non-depressed/depressed speakers using read aloud text containing constrained affective-linguistic criteria. Herein, using the Black Dog Institute Affective Sentences (BDAS) corpus, analysis demonstrates statistically significant feature differences in speech disfluencies, whereby when compared to non-depressed speakers, depressed speakers show relatively higher recorded frequencies of hesitations (55% increase) and speech errors (71% increase). Our study examines both manually and automatically labeled speech disfluency features, demonstrating that detailed disfluency analysis leads to considerable gains, of up to 100% in absolute depression classification accuracy, especially with affective considerations, when compared with the affect-agnostic acoustic baseline (65%).

Introduction

During mental health evaluations, it is standard practice for a clinician to evaluate a patient's spoken language behavior. Individuals suffering from depression disorders often exhibit psychogenic voice disturbances that adversely change their autonomic system and personality (Perepa, 2017). Recorded exemplars of speech-language disruptions in clinically depressed patients include disfluent speech patterns, abandonment of phrases, and unusually long response latencies (Breznitz & Sherman, 1987; Greden & Carroll, 1980; Hoffman et al., 1985). Patients with clinical depression also exhibit a greater number of speech hesitations (i.e. pauses, repeats, false-starts) than non-depressed populations during communication due to psychomotor agitation/retardation and cognitive processing delays (Alpert et al., 2001; Cannizzaro et al., 2004; Darby et al., 1984; Duffy, 2008; Ellgring & Scherer, 1996; Fossati et al., 2003; Hartlage et al., 1993; Nilsonne, 1987, Nilsonne et al., 1988; Szabadi et al., 1976).

Speech-based depression studies (Alghowinem et al., 2012; Alpert et al., 2001; Esposito et al., 2016; Mundt, 2012; Nilsonne et al., 1988; Stassen et al., 1998; Szabadi et al., 1976) have evaluated pause durations and frequency ratios (e.g. filled-pause rate, empty pause rate) with varied success. For example, Alghowinem et al. (2012) and Esposito et al. (2016) observed that average spontaneous speech pause durations were significantly longer in depressed speakers than non-depressed speakers. Due to unrestrained spontaneous speech variables, pause and rate ratios (i.e. the total number of pauses divided by the total number of words; the total pause time divided by the total recording time) have often been used to help compare utterances of different lengths (Liu et al., 2017).

Concerning speech errors, Rubino et al. (2011) discovered that depressed speakers exhibited significantly greater numbers of referential failures (i.e. including word replacement errors, such as malapropisms) than non-depressed speakers during spontaneous tasks. A malapropism is an incorrect substitution word for an intended word. By definition, a malapropism is unrelated in meaning and has a similar pronunciation, grammatical category, word stress, and syllable length (Fay & Cutler, 1977). Since Rubino et al. (2011), no automatic speech-based depression studies have pursued using speech errors as prospective discriminative depression features – and surprisingly, not even for predetermined read tasks.

Diagnostic applications based on automatic speech-based depression classification are often reliant on the uniformity of the patient elicitation procedure. Each elicitation method can include a wide range of linguistic structure, affect information, and task requirements (Howe et al., 2014; Stasak et al., 2017, 2018b). To date, there is no clinically approved speech-depression automatic diagnosis protocol for widespread use, and there is not even consensus on elicitation methods among researchers. Many exploratory speech-based depression techniques have been systematically investigated using a combination of speech-related features (i.e. acoustic, text-based), machine learning techniques, and speech elicitation protocols (see e.g.: Cummins et al., 2015; Jiang et al., 2017; Liu et al., 2017; Stasak et al., 2017; Valstar et al., 2016). However, still to date, no specific system to aid depression diagnosis holds dominance over all others.

The aim of experiments herein is to investigate speech disfluency behaviors in non-depressed/depressed speakers using read text containing specific affective-linguistic criteria. We hypothesize that unlike spontaneous speech, sentences containing target words within specific valence ranges will provide more accurate ground-truth for disfluency analysis as a result of their phonetic and affective constraints; further affording directly comparable speech data between different speakers. While natural speech disfluencies are common in spontaneous speech (Johnson et al., 2004; Gósy, 2003), due to the abnormal cognitive-motor effects of depression on cognitive skills (Breznitz & Sherman, 1987; Greden & Carroll, 1980; Hoffman, et al., 1985), we hypothesize that during simple sentence reading tasks depressed speakers will demonstrate greater numbers of abnormal pauses and speech errors than non-depressed speakers.

According to structural affect theory (Brewer & Lichtenstein, 1982) and information structure studies (Arnold et al., 2013, Dahan, 2015), the emotions of a reader can be systematically manipulated by the sensitive order in which information is presented in a text. Based the aforementioned studies, it is hypothesized that positioning the affective target word at the beginning of sentence rather than middle or end of the sentence will allow a speaker earlier cognitive cues for mood processing and appropriate prosodic phrasing. We suspect that healthy speakers will utilize initial sentence position affective keyword information differently than depressed speakers (i.e. less prosodic range or appropriateness (Cummins et al., 2015)).

In regards to the reading of texts, Salem et al. (2017) found that direct discourse (e.g. first-person) narratives elicited a stronger feeling of taking over the perspective of the character than indirect discourse (e.g. third-person) narratives. Therefore, it is anticipated that features extracted from first-person read narrative sentences will generate better classification results than third-person narrative sentences due to greater emotional attachment. We hypothesize that depressed speakers will exhibit less dynamic emotional vocal range behaviors than healthy speakers on account of their negative fixation (Goeleven et al., 2006; Gotlib & McCann, 1984) and/or less vocal control due to psychomotor agitation/retardation (Flint et al., 1993; Hoffman et al., 1985). Due to passive avoidance strategies exhibited by people with depression (Holahan & Moos, 1987; Holahan et al., 2005), it is also anticipated that depressed speakers will attempt fewer self-corrections after speech errors when compared with non-depressed speakers.

Due to the constraints of text-dependent stimuli, both manual and automatic speech recognition disfluency annotations disfluency attributes are evaluated. It is anticipated that text-dependent constraints will improve the precision of the ASR output since the acoustic phonetic variability in the elicited speech is smaller than that of, for example, spontaneous speech.