Multiple Model-Based Control Using Finite Controlled Markov Chains

Abstract

Cognition and control processes share many similar characteristics, and decisionmaking and learning under the paradigm of multiple models has increasingly gained attention in both fields. The controlled finite Markov chain (CFMC) approach enables to deal with a large variety of signals and systems with multivariable, nonlinear, and stochastic nature. In this article, adaptive control based on multiple models is considered. For a set of candidate plant models, CFMC models (and controllers) are constructed off-line. The outcomes of the CFMC models are compared with frequentist information obtained from on-line data. The best model (and controller) is chosen based on the Kullback–Leibler information. This approach to adaptive control emphasizes the use of physical models as the basis of reliable plant identification. Three series of simulations are conducted: to examine the performace of the developed Matlab-tools; to illustrate the approach in the control of a nonlinear nonminimum phase van der Vusse CSTR plant; and to examine the suggested model selection method for the adaptive control.

Appendix A: Kullback–Leibler Distance

Following Kulhavy [17], suppose that X ₁, X ₂, …, X _k+1 form a controlled Markov chain with a conditional probability mass function S(y|z,u) = Pr $\{X_{k+1}=y {\vert} X_{k}=z, U_{k}=u\}.$ The transition probability distribution is known partially, it is assumed to belong to a family S _θ. The task is to estimate θ.

For a sequence of observations ${\bf x}=\left( x_{1},x_{2},...,x_{k+1}\right)$ and control actions ${\bf u}=\left( u_{1},u_{2},...,u_{k+1}\right),$ we have

$$ \begin{aligned} S_{\theta }^{k}\left( {\bf x}|x_{1},{\bf u}\right) &=\prod_{i=1}^{k}S_{\theta }\left( x_{i+1}|x_{i},u_{i}\right)\\ &=\exp \left\{\sum_{i=1}^{k}\log S_{\theta }\left( x_{i+1}|x_{i},u_{i}\right) \right\}\\ &=\exp\left\{ k\sum_{\left( y,z,v\right) \in {\mathcal{X}}^{2}\times {\mathcal{U }}}R_{{\bf x,u}}\left( y,z,v\right) \log S_{\theta }\left( y|z,v\right) \right\}\\ &=\exp \left\{ -k\left[\overline{H}\left( R_{{\bf x}}\right) +\overline{D} \left( R_{{\bf x}}||S_{\theta }\right) \right] \right\} \end{aligned} $$

where

• $R_{\bf x,u}(a,b,c)$ is the empirical distribution $R_{{\bf x,u}}( a,b,c) =\frac{N_{\bf x,u}( a,b,c)}{k},$ and $N_{{\bf x,u}}(a,b,c)$ counts the number of occurrences of the triplets (a,b,c) in the sequence formed by ${\bf x}$ and ${\bf u}.$

• $\overline{H}(R_{{\bf x,u}})$ is the conditional Shannon entropy

  $\overline{H}\left(R_{{\bf x,u}}\right) =\sum_{\left( y,z,v\right) }R_{ {\bf x,u}}\left( y,z,v\right) \log R_{{\bf x,u}}\left( y,z,v\right) +\sum_{\left( z,v\right) }R_{{\bf x,u}}\left( z,v\right) \log R_{ {\bf x,u}}\left( z,v\right)$

  of a random variable Y given another random variable Z and a control action V, described jointly by a probability distribution $R_{{\bf x,u}},$ and

• $\overline{D}\left( R_{{\bf x,u}}||S_{\theta }\right)$ is the conditional Kullback–Leibler distance $\overline{D}\left( R_{{\bf x,u}}||S_{\theta }\right) =\sum_{\left( y,z,v\right) }R_{{\bf x,u}}\left( y,z,v\right) \log \frac{{R_{{\bf x,u}}}(y,z,v)}{S_{\theta}( y|z,v) R_{\bf x,u}(z,v)}$

• of a joint probability distribution $R_{{\bf x,u}}$ and a conditional distribution S _θ.

Further, we can regard any of the conditional distributions S _θ(y|z,v) as a set of distributions S ^z,v_θ (y), and conditional empirical distribution $R_{{\bf x,u}}\left( y|z,v\right)$ as a set of points $R_{{\bf x}}^{z,v}\left( y\right),$ $\% z\in {\mathcal{X}},$ $v\in {\mathcal{U}}.$ We can then write

$$ \overline{D}(R_{{\bf x,u}}||S_{\theta }) =\sum_{( z,v) \in {\mathcal{X}}\times {\mathcal{U}}}R_{{\bf x,u}}( z,v) D( R_{{\bf x,u}}^{z,v}||S_{\theta }^{z,v}). $$

The posterior distribution of the unknown parameter $\theta$ conditional on ${\bf x}$ and ${\bf u}$ is

$$ P_{{\bf x}}\propto P(\theta) S_{\theta }^{k}({\bf x} ,{\bf u}|x_{1}) \propto P(\theta) \exp \{ -k \overline{D}(R_{{\bf x,u}}||S_{\theta })\} $$

since the conditional entropy $\overline{H}(R_{{\bf x}})$ does not depend on θ.