Elsevier

Computer Speech & Language

Volume 34, Issue 1, November 2015, Pages 232-255
Computer Speech & Language

A hybrid approach to dialogue management based on probabilistic rules

https://doi.org/10.1016/j.csl.2015.01.001Get rights and content

Highlights

  • We present a new, hybrid modelling framework for dialogue management based on probabilistic rules.

  • The probabilistic rules function as high-level templates for the generation of a directed graphical model.

  • The rule parameters may be estimated from dialogue data via Bayesian inference.

  • The OpenDial toolkit allows system designers to develop dialogue systems using probabilistic rules.

  • User evaluation in a HRI domain shows that the approach outperforms traditional hand-crafted and statistical models.

Abstract

We present a new modelling framework for dialogue management based on the concept of probabilistic rules. Probabilistic rules are defined as structured mappings between logical conditions and probabilistic effects. They function as high-level templates for probabilistic graphical models and may include unknown parameters whose values are estimated from data using Bayesian inference. Thanks to their use of logical abstractions, probabilistic rules are able to encode the probability and utility models employed in dialogue management in a compact and human-readable form. As a consequence, they can reduce the amount of dialogue data required for parameter estimation and allow system designers to directly incorporate their expert domain knowledge into the dialogue models.

Empirical results of a user evaluation in a human–robot interaction task with 37 participants show that a dialogue manager structured with probabilistic rules outperforms both purely hand-crafted and purely statistical methods on a range of subjective and objective quality metrics. The framework is implemented in a software toolkit called OpenDial, which can be used to develop various types of dialogue systems based on probabilistic rules.

Introduction

The design of dialogue strategies is a challenging task in the development of spoken dialogue systems (SDS). The selection of system actions is often grounded in a complex dialogue state encompassing a variety of factors such as the dialogue history, the user goals and preferences, the external context and the task to perform. In addition, spoken dialogue is also riddled with uncertainties arising from speech recognition errors, ambiguous inputs, partially observable environments, and unpredictable dialogue dynamics. These difficulties are particularly striking in the case of human–robot interaction (HRI). By their very definition, human–robot interactions take place in a physical, situated environment that must be captured and monitored by the robotic agent. They must also typically deal with high levels of noise and uncertainty caused by e.g. imperfect sensors and actuators. The robot's tracking of the current dialogue state is therefore bound to remain partial and error-prone.

Two families of dialogue management approaches have been historically developed to address these issues. The first family relies on hand-crafted strategies, ranging from finite-state automata to more complex inference procedures based on formal logic and classical planning. These strategies provide principled techniques for the interpretation and generation of dialogue moves on the basis of the dialogue participants’ mental states (including their shared knowledge). Dialogue is then framed as a collaborative activity in which the interlocutors work together to coordinate their actions, maintain a shared conversational context, resolve open issues and satisfy social obligations (Allen et al., 2000, Larsson, 2002, Jokinen, 2009). Such approaches can yield detailed analyses of various dialogue behaviours, but they generally assume complete observability of the dialogue state and provide only a limited account of errors and uncertainties. In addition, the knowledge bases from which the system's decisions are derived must be completely specified in advance by domain experts. Their deployment in practical applications is thus non-trivial.

The second family relies on statistical modelling techniques (Levin et al., 2000, Roy et al., 2000, Young et al., 2010, Rieser and Lemon, 2011). The dialogue is here represented as a stochastic control process – often a Markov decision process (MDP) or a Partially observable Markov decision process (POMDP) – and the optimal dialogue strategy is the one that maximises the system's long-term expected utility. These probabilistic models offer an explicit account for the various uncertainties that can arise during the interaction. They also allow the dialogue strategies to be optimised in a data-driven manner instead of relying on hand-crafted mechanisms, making it easier to adapt to new environments or users. However, these probabilistic models typically depend on large amounts of training data to estimate their parameters – a requirement that is hard to satisfy for most dialogue domains. This shortage of relevant datasets is especially critical in human–robot interactions, given the high costs of collecting and annotating dialogue data for these dialogue domains.

This article presents a hybrid approach to dialogue management that seeks to combine the benefits of hand-crafted and statistical techniques in a single framework. As in previous work on POMDPs models for dialogue management, the approach represents the dialogue state as a Bayesian network that is regularly updated with new observations and employed to derive the system's actions. However, the domain models are no longer expressed with traditional factored representations but are instead structured via probabilistic rules. As explained in the next pages, the rules can be viewed as high-level templates for probabilistic graphical models. The use of probabilistic rules provides an efficient abstraction layer that allows the system designer to capture the domain models in a concise and human-readable form.

The present article is structured as follows. Section 2 reviews the key principles of dialogue management, focusing in particular on MDP- and POMDP-based approaches. Section 3 outlines the formalism of probabilistic rules and their instantiation as nodes of a graphical model. Section 4 describes how the parameters of probabilistic rules can be estimated via Bayesian inference, using either Wizard-of-Oz data (supervised learning) or user interactions (reinforcement learning). Section 5 presents the OpenDial toolkit, a domain-independent dialogue toolkit that allows dialogue developers to construct dialogue systems using probabilistic rules. Section 6 describes a user evaluation of this modelling approach in a human–robot interaction domain. Section 7 contrasts the framework with related work. Finally, Section 8 concludes the article and reviews future research directions.

Section snippets

System architecture

The general architecture of a spoken dialogue system is depicted in Fig. 1. The user speech signals are first processed by the speech recogniser, resulting in a list of recognition hypotheses u˜u, where each hypothesis is associated to a particular probability or confidence score.1 Dialogue understanding then maps these hypotheses into

Probabilistic rules

One major bottleneck in statistical approaches to dialogue management is the size of the parameter space. The framework presented in this article seeks to reduce the numbers of parameters by taking advantage of expert knowledge about the dialogue domain. More precisely, the framework rests on the idea of representing the transition and utility models of a dialogue POMDP in terms of probabilistic rules. These rules are practically defined as if...then...else constructions that map logical

Parameter estimation

The examples of probabilistic rules shown in the previous section relied on probabilities and utilities fixed by hand. In practical dialogue domains, such parameters are often difficult to determine manually, due to the inherent unpredictability that characterise spoken dialogue. They are therefore best estimated empirically from actual dialogue data. We developed two alternative methods for parameter estimation: a supervised learning approach based on Wizard-of-Oz data, and a reinforcement

Implementation

The framework presented in this article is fully implemented in an open-source software toolkit called OpenDial.5 The toolkit, which is released under an MIT license, is a Java-based, domain-independent platform for the development of spoken dialogue systems based on the specification of probabilistic rules in XML format.

Listing 1 illustrates an example of a probability rule (corresponding to rule r2 in Section 3)

User evaluation

A user experiment was conducted to evaluate the practical performance of probabilistic rules compared to existing modelling approaches. The aim of the experiment was to compare three alternative approaches to dialogue management in a human–robot interaction domain. The three approaches respectively correspond to:

  • a purely hand-crafted approach, based on a finite-state automaton

  • a purely statistical approach, based on factored statistical models

  • a hybrid approach based on probabilistic rules

The

Discussion and related work

The use of structural knowledge in probabilistic models is a recurring theme in the fields of reinforcement learning (Hauskrecht et al., 1998, Pineau, 2004, Kersting and Raedt, 2004, Lang and Toussaint, 2010, van Otterlo, 2012), and statistical relational learning (Jaeger, 2001, Richardson and Domingos, 2006, Getoor and Taskar, 2007). This structure may take a hierarchical or relational form. As in the probabilistic rules presented in this article, most of these frameworks rely on the use of

Conclusion

This article presented a hybrid approach to dialogue management that combines statistical modelling with expert domain knowledge. As in existing POMDP approaches to dialogue management, the dialogue state is represented as a Bayesian network that is regularly updated with observations and used to derive high-utility system actions. The internal models of the dialogue domain are, however, represented via probabilistic rules. At runtime, the probabilistic rules for the domain are instantiated as

Acknowledgements

The research presented in this article was funded by a Ph.D. Research Fellowship from the University of Oslo and a Post-doctoral Research Grant from the Norwegian Research Council. The author wishes to thank Stephan Oepen, Erik Velldal, Geert-Jan M. Kruijff, Kristiina Jokinen, Verena Rieser and Andrei Popescu-Belis for many useful comments.

References (58)

  • H.R. Chinaei et al.

    Learning observation models for dialogue POMDPs

  • D.D. Corkill

    Design alternatives for parallel and distributed blackboard systems

    Blackboard Architectures and Applications

    (1989)
  • P.A. Crook et al.

    Representing uncertainty about complex user goals in statistical dialogue systems

  • H. Cuayáhuitl

    Learning dialogue agents with Bayesian relational state representations

  • L. Daubigney et al.

    Off-policy learning in large-scale POMDP-based dialogue systems

  • F. Doshi et al.

    Spoken language interaction with model uncertainty: an adaptive human–robot interaction system

    Connect. Sci.

    (2008)
  • M. Frampton et al.

    Recent research advances in reinforcement learning in spoken dialogue systems

    Knowl. Eng. Rev.

    (2009)
  • R.M. Fung et al.

    Weighing and integrating evidence for stochastic simulation in Bayesian networks

  • M. Gašić et al.

    On-line policy optimisation of Bayesian spoken dialogue systems via human interaction

  • M. Gašić et al.

    On-line policy optimisation of spoken dialogue systems via live interaction with human subjects

  • L. Getoor et al.

    Introduction to Statistical Relational Learning

    (2007)
  • J. Ginzburg

    The Interactive Stance

    (2012)
  • M. Hauskrecht et al.

    Hierarchical solution of Markov decision processes using macro-actions

  • P. Heeman

    Combining reinforcement learning with information-state update rules

  • J. Henderson et al.

    Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets

    Comput. Linguist.

    (2008)
  • M. Jaeger

    Complex probabilistic modeling with recursive relational Bayesian networks

    Ann. Math. Artif. Intell.

    (2001)
  • K. Jokinen

    Constructive Dialogue Modelling: Speech Interaction and Rational Agents

    (2009)
  • K. Kersting et al.

    Logical Markov decision programs and the convergence of logical TD(lambda)

  • W.B. Knox

    Learning from Human-generated Reward

    (2012)
  • Cited by (51)

    • An empirical assessment of deep learning approaches to task-oriented dialog management

      2021, Neurocomputing
      Citation Excerpt :

      Also [29] proposed a different hybrid approach to dialog modeling in which n-best recognition hypotheses are weighted using a mixture of expert knowledge and data-driven measures by using an agenda and an example-based machine translation approach, respectively. Hybrid approaches to DM combine statistical and rule-based approaches to try to reduce the amount of dialog data required for parameter estimation and to allow system designers to directly incorporate their expert domain knowledge into the dialog management models [30]. Recently, encouraged by the positive results obtained in related areas, deep learning has started to be used for conversation [31].

    • A Cognitive Assistant for improving human reasoning skills

      2018, International Journal of Human Computer Studies
      Citation Excerpt :

      ControlUnit is the dialog manager of the Cognitive Assistant LIZA. It pursues the finite state-based approach (Lison, 2015) and implements the dialog flow illustrated by Fig. 3. After initiating a smalltalk with greetings and introduction, the ControlUnit creates a list of stories retrieved from the Storystore and uses it to guide the user through the training process.

    • Information-gathering dialog system using acoustic features and user’s motivation

      2023, IAES International Journal of Artificial Intelligence
    • Intent Detection Using Contextualized Deep SemSpace

      2023, Arabian Journal for Science and Engineering
    View all citing articles on Scopus

    This paper has been recommended for acceptance by R.K. Moore.

    View full text