Keywords

1 Introduction

Mental issues are increasing around the world and access to healthcare programs are limited. Internet-based interventions provide additional access and can close the gap between treatment and demand [8]. In these interventions, participants often provide diary data in which they rank, for example, their mood levels and simultaneously report daily activities. Because various activities from walking a dog, to volunteering, cleaning the house, or having a drink out with friends affect mood in different and complex ways [9], we attempt to analyze the effects that different activities have on the mood level.

In this study, we propose a two-step approach for the analysis of free text diary data that is provided by participants of an online depression treatment [4]. The dataset consists of 440 patients who provided 9,192 diary entries. We utilize text-mining techniques in order to categorize the free text into defined activity categories (exercise, sickness, rumination, work related, recreational, necessary, social, and sleep related activities) and use individualized partial ordered logit models to predict the mood level. This two-step approach allows for interpretability of the effects between the activity categories and the mood level. Thus, besides studying these relationships, we contribute to the field of machine learning by proposing a mixed method approach to analyze diary data. This short paper is based on a full paper already published in [3]. Here, more information about the methods, results, and discussion including a full list of references can be found.

2 Method

Figure 1 illustrates the two-step approach. In the first step, we utilize bag-of-words (BoW) categorization and extent the results by applying recurrent neuronal networks (RNN) [5] in order to categorize the free text into activity categories. We split all diary entries into sentences and identify the most frequent (\(\ge \)10 occurrences) 1- and 2-grams. Next, two of the authors manually associated the frequent 1- and 2-grams with an activity category. Only the 1- and 2-grams that are assigned identically by both authors are utilized for the BoW categorization. The sentences are then assigned to one or multiple activities based on the categorized n-grams. Since 8,032 sentences do not contain any of the n-grams, they cannot be categorized. We then train an Elman network (RNN) on the categorized sentences. The RNN classifies sentences that are not already assigned by the BoW categorization. Some sentences are not associated because these consist of words that do not appear in the training corpus. The results of the BoW categorization and the merged results of both approaches are then utilized as input for the second step.

Fig. 1.
figure 1

The process of the two-step approach.

Full size image

Because the mood level is ranked on a scale from one to ten, we use a partial ordered logit model for the prediction and the analysis of the effects between the assigned activity categories and the mood level. The ordered logit model is based on the proportional odds assumption (POA), which means that independent variables have the same effect on the outcome variable across all ranks of the mood level [7]. The partial ordered logit model, however, allows variables that violate this assumption to vary among the ranks. We test the assumption by a likelihood ratio test. The logit is then calculated as follows:

$$\begin{aligned} \begin{aligned} \ln (\theta _{ijt}) = \alpha _{ij}&- \left( \underbrace{\sum _{a \in A_1}^{} \beta _{aj} \, x_{ajt}}_{\text {POA holds}}\ + \underbrace{\sum _{a \in A_2}^{} \beta _{aij} \, x_{ajt}}_{\text {POA violated}}\right) , \end{aligned} \end{aligned}$$

where \(\alpha _{ij}\) represents the threshold between the ranks of the mood level for \(i=1, \ldots , I = 9\) and \(j=1, \ldots , J = 440\). The activities of participant j at time t are represented by \(x_{ajt}\), where \(A_1 =\) {sleep related, recreational activities} and \(A_2 =\) {exercise, sickness, rumination, social, work related, necessary activities}. The parameters to be estimated are \(\beta _{[...]}\). The index j in \(\alpha _{ij}\) addresses the problem of scale usage heterogeneity [6]. Additionally, we hypothesize that the effects of the activities vary among participants. Thus, we also include client specific \(\beta \)-parameters. For a robustness check, we also implement the partial ordered logit model without the consideration of heterogeneity among the participants (Model 1), only implement the individual \(\alpha \)-parameters (Model 2), only client specific \(\beta \)-terms (Model 3), and the above specified model including both heterogeneity terms (Model 4). Therefore, we obtain four different models, which we compare regarding their predictive performance.

3 Results and Discussion

We compare the models by using the Deviance Information Criterion (DIC), which is especially suited for Bayesian models that are estimated by MCMC methods [2]. The results of the DIC indicates a superior performance for the model that includes both heterogeneity terms. According to [1], however, the DIC can be prone to select overfitted models. Thus, for applying an out-of-sample test, we randomly extract mood entries (680 sentences) and their corresponding activities from the data before training the model. We then predict the mood level of the individuals in the test data and utilize the Root Mean Square Error (RMSE) as well as the Mean Absolute Error (MAE) as performance indicators. We also report performance measures for a so called Mean Model; here, we use the average mood level of the training set as predictions for the test dataset (in this case the mood level 6).

Table 1. Model comparison with levels of heterogeneity for each text-mining approach.
Full size table

As illustrated in Table 1, an increasing degree of heterogeneity reduces the prediction error. The additionally classified activities by the RNN do not contribute to an increased performance. This can potentially arise because the training data used for the RNN, which is based on the BoW categorization, might not be accurate enough for the RNN to generate new knowledge. Model 4 for the BoW categorization shows the best predictive performance. Thus, we utilize this model for revealing the relationships between the activities and the mood level.

We find that the category sickness has a strong negative and significant effect on mood. Furthermore, our analysis suggests that the category rumination affects the mood level in a negative way and social activities have a positive effect on the mood level. The other activities are not significant. These results are consistent with literature in the field [9]. During the ECML, we will additionally present the results of a model that directly predicts the mood levels based on the free text data.