A multi-modal dialogue analysis method for medical interviews based on design of interaction corpus

Abstract

We propose a multi-modal dialogue analysis method for medical interviews that hierarchically interprets nonverbal interaction patterns in a bottom-up manner and simultaneously visualizes the topic structure. Our method aims to provide physicians with the clues generally overlooked by conventional dialogue analysis to form a cycle of dialogue practice and analysis. We introduce a motif and a pattern cluster in the designs of the hierarchical indices of interaction and exploit the Jensen–Shannon divergence (JSD) metric to reduce the number of usable indices. We applied the proposed interpretation method of interaction patterns to develop a corpus of interviews. The results of a summary reading experiment confirmed the validity of the developed indices. Finally, we discussed the integrated analysis of the topic structure and a nonverbal summary.

Appendix

1.1 Motif occurrence

The normalized expected occurrence of motif Mot = (Pat₁,…,Pat _n ), assuming that all patterns are independent of each other, is as follows:

$$ Q^{\prime}_{d} \left( {\text{Mot}} \right) = {\frac{{{\frac{{P_{1} \ldots P_{N} }}{{\overline{P}_{2} \cdots \overline{P}_{N - 1} }}}\left( {{\frac{1}{{\overline{P}_{1} }}} + \cdots + {\frac{1}{{\overline{P}_{N} }}}} \right)}}{{L\left( {\text{Mot}} \right)}}}. $$

where P(Pat _i ) = P _i , $1 - P_{i} = \overline{{P_{i} }}$. $P^{\prime}_{d} \left( {\text{Mot}} \right)$ is given as motif occurrence $P_{d} \left( {\text{Mot}} \right)$ divided by L(Mot):

$$ P^{\prime}_{d} ({\text{Mot}}) = {\frac{{P_{d} ({\text{Mot}})}}{{L({\text{Mot}})}}}. $$

1.2 Pattern co-occurrence measure

When motif MOT _ij contains consecutive patterns Pat _i and Pat _j as its elements, as ${\text{MOT}} \supseteq {\text{MOT}}_{ij} = \left\{ {{\text{Mot}} = \left( { \ldots ,P_{i} ,P_{j} , \ldots } \right)} \right\},$ we can compute pattern co-occurrence measure ${\text{Cooc}}\left( {} \right)$ between patterns Pat _n and Pat _m :

$$ {\text{Cooc}}\left( {{\text{Pat}}_{n} ,{\text{Pat}}_{m} } \right) = \sum\limits_{{{\text{Mot}} \in {\text{MOT}}_{nm} }} {{\text{IBM}}_{D} \left( {\text{Mot}} \right)} + \sum\limits_{{{\text{Mot}} \in {\text{MOT}}_{mn} }} {{\text{IBM}}_{D} \left( {\text{Mot}} \right)} . $$