Travel Agency Task Dialogue Corpus: A Multimodal Dataset with Age-Diverse Speakers

Task-oriented dialogue systems have long been features in the study of dialogue [3, 4, 23, 36]. Recently, deep neural networks have come to be successfully applied to response generation [11, 29, 33] and dialogue state tracking [9, 19, 35]. These studies mainly focus on the generation of an appropriate response to user inputs.

When individuals communicate, they use different vocabularies, speaking speeds, facial expressions, and gestural languages, depending on those with whom they are communicating. For example, with children, speakers may use simpler vocabulary and speak with more emotion, whereas with the elderly, people may speak more slowly. By contrast, dialogue systems currently in use rarely change their speaking style or dialogue strategy according to the user. We believe that dialogue systems should alter their dialogue strategies in response to the identity of the user to accomplish their tasks more efficiently and increase user satisfaction.

A range of factors, such as gender, social relationships, and roles, significantly affect communication. However, here, we focused on the age of the speaker as a factor that dramatically affects communication, as speakers of various ages are relatively easy to recruit, and age is among the most important factors among them. We collected a multimodal dialogue corpus including various speaker ages from children to the advanced age (Figure 1).

For the dialogue topics, we focused on travel, as it interests people of all ages. We created a task based on a tourism consultation between an operator and a customer at a travel agency. During these dialogues, the operator was enabled to use a tourist information retrieval system to obtain information on tourist spots. We collected information on timestamps, queries, and responses. We also manually transcribed the dialogues and annotated them, including noting tourist spots mentioned and the queries sent to the tourist system, along with their place in the dialogue. These data helped construct dialogue systems to access external resources during interactions.

The features of our corpus are as follows:

It is generally unrealistic to have children to consulting travel agencies. However, children interacting with a dialogue system that provides tourism advice can be realistic. This corpus can be use as training data for constructing such a system. In addition, considering that the dialogue style can differ significantly between adults and minors, this corpus is valuable for training and evaluating dialogue models for adaptation to diverse dialogue strategies. In Section 6 of this study, the ways in which dialogue strategies varied among the minor, adult, and elderly participants can be illuminating. In Section 8, we conducted dialogue act prediction experiments to demonstrate the usefulness of this corpus.

This article describes the details of the dialogue task, the methods and results of the corpus collection, and the results of the analysis of the collected corpus with respect to the dialogue phase transition and the customers’ facial expressions to identify an effective interaction strategy with respect to the user’s age for constructing dialogue systems.

This article forms an updated and rectified version of a paper presented at a conference [13]. The conference proceedings included only one type of annotation (general dialogue act) and analysis based on the annotation. The revisions of the original manuscript include the addition of three novel types of annotation (task-specific dialogue act, query, and tourist spot references), along with new analyses and experiments enabled by the new annotations.

Several multimodal dialogue corpora including dialogues between two speakers have been collected for the analysis of human interactions, facial expressions, emotions, and gestures. The Cardiff Conversation Database (CCDb) [1] contains audio-visual natural conversation with no role assigned (whether listener or speaker) and no scenario. Some of the data were annotated with dialogue acts such as Backchannel and Agree, emotions such as Surprise and Happy, and head movements such as Head Nodding and Head Tilt. Each conversation lasted 5 minutes, and 300 minutes of dialogue were collected, with participants ranging in age from 25 to 57 years. The Emotional Dyadic Motion CAPture (IEMOCAP) dataset [6] is used for communication and gesture analysis. The actors were wearing markers on their faces, heads, and hands, and two types of dialogue were conducted: improvisations and scripted scenarios. The utterances are annotated with emotional labels. The total recording time was approximately 12 hours. The NoXi corpus [7] contains dialogues mainly in English, French, and German, annotated with head movements, smiles, gazes, engagement, and so on. The total recording time was approximately 25 hours, and the ages of the participants ranged from 21 to 50 years. The CANDOR corpus [24] is an English conversation corpus that consists of 1,656 conversations totaling 850 hours. This corpus is larger than that collected for this study. However, unlike our study, it was transcribed and annotated automatically, not by hand. In addition, the participants range in age from 19 to 66 years, and the corpus excludes conversations with minors. The CABB dataset [12] is a dialogue corpus of collaborative referential communication games conducted in Dutch. It includes video, audio, and body-motion tracking data. The multimodal dialogue corpus Hazumi [16] includes dialogues that represent conversations in Japanese between a participant and a system that is operated using the Wizard of Oz (WoZ) method. It aggregates approximately 65 hours of dialogue¹ with 214 participants who ranged in age from their 20s to their 70s, excluding minors. All exchanges between pairs, made up of a system utterance and the subsequent user utterance, were assigned sentiments by multiple third-party annotators. Physiological sensor outputs were also simultaneously recorded as a version within the corpus. In addition, a multimodal corpus of persuasive dialogues between participants and an Android operated with WoZ has been constructed [15]. In this study, we collected more than 115 hours of data, or more than the two-party multimodal dialogue corpus noted previously. The age range of the speakers in our data was also wider than that in previous studies.

Multimodal corpora that contain conversations between multiple people have also been collected. RoomReader [25] is a multi-party, multimodal dialogue corpus that consists of approximately 8 hours of dialogue that were collected via Zoom. The dialogues were transcribed automatically and manually corrected, with annotations made regarding engagement. The Belfast storytelling dataset [17], the AMI meeting corpus [8],the ICSI meeting corpus [14], Computers in Human Interaction Loop (CHIL) [27], and Video Analysis and Content Extraction (VACE) [10] are well-known examples of this. These corpora contain both two-person and multi-person dialogues, primarily encompassing speakers in their 20s to 70s and generally excluding minor speakers.

Several multimodal dialogue datasets were constructed through the extraction of scenes from TV series. The Understanding and Response Prediction dataset [28] is a multimodal dialogue corpus consisting of approximately 42,000 scenes from TV dramas. This corpus was constructed for the boundary prediction of dialogue scenes and response generation from scene recognition. The Multimodal EmotionLines Dataset (MELD) [21] is a dialogue emotion recognition dataset that was created through the annotation of emotional information in dialogue scenes from the television show Friends. Another dialogue emotion recognition dataset, the Multimodal Multi-scene Multi-label Emotional Dialogue (M³ED) dataset [34], was constructed from Chinese TV series.

Monologue corpora include speakers of a broader age range. CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) [32] is one of the largest, containing 23,500 YouTube videos of 1,000 people, each utterance of which is annotated with an emotion label. In addition, several monologue corpora have been shared in the research community, such as the Multimodal Corpus of Sentiment Intensity (CMU-MOSI) [31], the ICT Multi-Modal Movie Opinion (ICT-MMMO) [30], and the Multimodal Opinion Utterances Dataset (MOUD) [20]. These datasets are relatively large, and they include minor and older speakers; however, they are not dialogue corpora.

We collected dialogues between two speakers, where one was playing the role of an operator and the other was playing the role of a customer, simulating a travel agency consultation. The two speakers made a video call using Zoom,² and each conversation lasted 20 minutes. The customer prepared a travel concept before the beginning of the dialogue (described in Section 3.2) and then consulted with the operator to identify tourist spots that met the conception. The operator elicited the customer’s request and recommended tourist spots using information obtained from the tourist information retrieval system (described in Section 3.1).

The collected dialogues were manually transcribed, and we annotated the transcriptions with the following four types of annotation: (1) the subset of the ISO 24617-2 dialogue act tag (general dialogue act), (2) task-specific dialogue act tag for our undertaking (task-specific dialogue act), (3) mapping tourist spots mentioned by speakers against tourist spot IDs registered in the tourist information retrieval system, and (4) plotting search queries with functional segments in the dialogues.

The functional segments are defined in the ISO 24617-2 annotation scheme as units that are smaller than that of the utterances [5]. We performed all four types of annotation on functional segments rather than on utterances.

The sections that follow describe the procedure that we adopted to classify utterances into functional segments and subsequently elucidate all of the annotations performed. For each type of annotation, one annotator was assigned to perform annotations for a particular dialogue. We verified the consistency of annotations through experiments, which are also elucidated in the ensuing sections.

In the following sections, we analyze the verbal and non-verbal information to determine how the speakers’ behavior differs between age groups. One of the most commonly used methods for visualizing the time structure of sequential data is the hidden Markov model (HMM) [18]. In this section, we describe training HMM using sequences of dialogue act and discuss differences in dialogue transition among age groups to analyze verbal information.

For non-verbal information, we investigated the difference in the operator’s facial expression between customers’ age groups; we cropped regions of the operator’s face from the video images and extracted facial action units (AUs) using OpenFace [2]. The facial regions were too small to extract AUs in half of the data because the dialogues were conducted while sharing the screen, so we used 165 dialogues recorded with gallery view for the analysis.

The AUs were obtained frame by frame, and they were averaged over the dialogue to compare age groups. We conducted a one-way layout ANOVA factoring the age group and performed multiple comparison tests for AUs that showed significant differences. In the analysis, we focused on the AUs relating to the smile. Table 12 shows the results of multiple comparison tests for AU06 (Cheek raiser) and AU12 (Lip corner puller), which exhibited a significant difference from ANOVA. These AUs become high when the speaker expressed a smile.

As shown in the table, significant differences were observed between the minor group and the adult group for AU06 and AU12. For both AUs, the values were larger in the minor’s group. These results reflect that the operator expressed a smile more frequently to the minor customer. The system should therefore adapt not only verbal but also non-verbal dialogue behavior to the user’s age to provide natural travel guidance.

This work represents our multimodal dialogue corpus with various speakers’ ages, from children to those of advanced age. This corpus includes 330 dialogues of 20 minutes each. The dialogue task given was based on the consultation of tourist spots at a travel agency between two speakers, one of whom played the role of a tourism services operator and the other a customer. The dialogues were manually transcribed and annotated using four types of annotations: general dialogue act, task-specific dialogue act, query, and tourist spot mention.

The number of individuals in each age group included in this corpus is uneven, with data from older speakers being relatively scarce. In future work, we plan to expand our corpus to include age groups that are currently underrepresented. We also plan to analyze dialogues taking into account cultural backgrounds and personalities, and to develop a dialogue model that adapts its strategy based on user characteristics.

Our corpus has been distributed by the Informatics Research Data Repository at the National Institute of Informatics (NII-IDR).⁹

The authors would like to thank Koki Washio for useful discussions.

The task-specific dialogue act tags can be categorized into those assigned to operator segments and those assigned to customer segments. Tables 14 and 15 display and define every tag along with sample segments.