Ad hoc teamwork by learning teammates’ task

Abstract

This paper addresses the problem of ad hoc teamwork, where a learning agent engages in a cooperative task with other (unknown) agents. The agent must effectively coordinate with the other agents towards completion of the intended task, not relying on any pre-defined coordination strategy. We contribute a new perspective on the ad hoc teamwork problem and propose that, in general, the learning agent should not only identify (and coordinate with) the teammates’ strategy but also identify the task to be completed. In our approach to the ad hoc teamwork problem, we represent tasks as fully cooperative matrix games. Relying exclusively on observations of the behavior of the teammates, the learning agent must identify the task at hand (namely, the corresponding payoff function) from a set of possible tasks and adapt to the teammates’ behavior. Teammates are assumed to follow a bounded-rationality best-response model and thus also adapt their behavior to that of the learning agent. We formalize the ad hoc teamwork problem as a sequential decision problem and propose two novel approaches to address it. In particular, we propose (i) the use of an online learning approach that considers the different tasks depending on their ability to predict the behavior of the teammate; and (ii) a decision-theoretic approach that models the ad hoc teamwork problem as a partially observable Markov decision problem. We provide theoretical bounds of the performance of both approaches and evaluate their performance in several domains of different complexity.

Appendix: Proofs

In this appendix, we collect the proofs of the different statements introduced in the main text.

1.1 Proof of Theorem 1

In our proof, we use the following auxiliary result, which is a simple extension of Lemma 2.3 in [12].

Lemma 1

For all $N\ge 2$, for all $\beta \ge \alpha \ge 0$ and for all $d_1,\ldots ,d_N\ge 0$ such that

$$\begin{aligned} \sum _{i=1}^Ne^{-\beta d_i}\ge 1, \end{aligned}$$

let

$$\begin{aligned} q_i=\frac{e^{-\beta d_i}}{\sum _{j=1}^Ne^{-\beta d_j}}. \end{aligned}$$

Then, for all $x_1,\ldots ,x_N$,

$$\begin{aligned} \ln \frac{\sum _{i=1}^Ne^{-\alpha d_i+x_i}}{\sum _{j=1}^Ne^{-\beta d_i}} \le \frac{\beta -\alpha }{\beta }\ln N+\mathbb {E}_{}\left[ X\right] , \end{aligned}$$

where $X$ is a r.v. taking values in $\left\{ x_1,\ldots ,x_N\right\}$ and such that $\mathbb {P}_{}\left[ X=x_i\right] =q_i$.

Proof

We have

$$\begin{aligned} \ln \frac{\sum _{i=1}^Ne^{-\alpha d_i+x_i}}{\sum _{j=1}^Ne^{-\beta d_i}}&=\ln \frac{\sum _{i=1}^Ne^{-\beta d_i}e^{-(\alpha -\beta )d_i+x_i}}{\sum _{j=1}^Ne^{-\beta d_i}}\nonumber \\&=\ln \mathbb {E}_{}\left[ e^{-(\alpha -\beta )D+X}\right] \nonumber \\&\le (\beta -\alpha )\mathbb {E}_{}\left[ D\right] +\mathbb {E}_{}\left[ X\right] , \end{aligned}$$

(17)

where $D$ is a r.v. taking values in $\left\{ d_1,\ldots ,d_N\right\}$ such that $\mathbb {P}_{}\left[ D=d_i\right] =q_i$, and the last inequality follows from Jensen’s inequality. Since $D$ takes at most $N$ values, we have that

$$\begin{aligned} H(D)\le -\sum _{i=1}^N\frac{1}{N}\ln \frac{1}{N}=\ln N, \end{aligned}$$

where $H(D)$ is the entropy of the r.v. $D$, and hence

$$\begin{aligned} \ln N&\ge H(D)\nonumber \\&=\sum _{i=1}^Nq_i\left( \beta d_i+\ln \sum _{j=1}^Ne^{-\beta di}\right) \nonumber \\&=\beta \mathbb {E}_{}\left[ D\right] +\ln \sum _{j=1}^Ne^{-\beta di}\nonumber \\&\ge \beta \mathbb {E}_{}\left[ D\right] . \end{aligned}$$

(18)

Finally, because $\beta \ge \alpha$, we can replace (18) in (17) to get

$$\begin{aligned} \ln \frac{\sum _{i=1}^Ne^{-\alpha d_i+x_i}}{\sum _{j=1}^Ne^{-\beta d_i}}&\le \frac{\beta -\alpha }{\beta }\ln N+\mathbb {E}_{}\left[ X\right] . \end{aligned}$$

$\square$

We are now in a position to prove Theorem 1. Let $h_{1:n}$ be a fixed history. Define $\gamma _0=0$ and, for $t\ge 0$, let

$$\begin{aligned} w_t(\tau )&=e^{-\gamma _tL_\tau (h_{1:t})},&W_t&=\sum _{\tau \in \mathcal {T}}w_t(\tau ),&Q_t(\tau )&=\frac{w_t(\tau )}{W_t}. \end{aligned}$$

Using the notation above, we have that

$$\begin{aligned} P(h_n,a)&=\sum _{\tau \in \mathcal {T}}\frac{w_n(\tau )}{W_n}E_\tau (h_{1:n},a)\\&=\mathbb {E}_{\tau \sim Q_n}\left[ E_\tau (h_{1:n},a)\right] . \end{aligned}$$

Our proof closely follows the proof of Theorem 2.2 of [12]. We establish our result by deriving upper and lower bounds for the quantity $\ln (W_n/W_0)$.

On the one hand,

$$\begin{aligned} \ln \frac{W_n}{W_0}=\ln \left( \sum _{\tau \in \mathcal {T}}w_n(\tau )\right) -\ln \left|\mathcal {E}\right|. \end{aligned}$$

Becayse $w_t(\tau )>0$ for all $t$, then for any $\tau \in \mathcal {T}$,

$$\begin{aligned} \ln \frac{W_n}{W_0} \ge \ln w_n(\tau )-\ln \left|\mathcal {E}\right|=-\gamma _nL_\tau (h_{1:n})-\ln \left|\mathcal {E}\right|. \end{aligned}$$

(19)

On the other hand, for $t>1$,

$$\begin{aligned} \ln \frac{W_t}{W_{t-1}}&=\ln \left( \frac{\sum _{\tau }e^{-\gamma _tL_\tau (h_{1:t})}}{\sum _{\tau }e^{-\gamma _{t-1}L_\tau (h_{1:t-1})}}\right) \\&=\ln \left( \sum _{\tau }\frac{e^{-\gamma _tL_\tau (h_{1:t-1})-\gamma _t\ell _\tau (h_{1:t})}}{\sum _{\tau }e^{-\gamma _{t-1}L_\tau (h_{1:t-1})}}\right) . \end{aligned}$$

Using Lemma 1, we can write

$$\begin{aligned} \ln \frac{W_t}{W_{t-1}}&\le \frac{\gamma _{t-1}-\gamma _t}{\gamma _{t-1}}\ln \left|\mathcal {E}\right|-\gamma _t\mathbb {E}_{\tau \sim Q_{t-1}}\left[ \ell _\tau (h_{1:t})\right] \\&=\frac{\gamma _{t-1}-\gamma _t}{\gamma _{t-1}}\ln \left|\mathcal {E}\right|-\gamma _t\ell _P(h_{1:t}), \end{aligned}$$

where the last equality follows from the definition of $P$. Noting that

$$\begin{aligned} \ln \frac{W_1}{W_0}\le -\gamma _t\ell _P(h_1), \end{aligned}$$

we can set the upper bound $\ln (W_n/W_0)$ as

$$\begin{aligned} \ln \frac{W_n}{W_0}&\le \sum _{t=1}^n\ln \frac{W_t}{W_{t-1}}\\&=\sum _{t=2}^n\frac{\gamma _{t-1}-\gamma _t}{\gamma _{t-1}}\ln \left|\mathcal {E}\right|-\sum _{t=1}^n\gamma _t\ell _P(h_{1:t}). \end{aligned}$$

The fact that $\gamma _{t+1}<\gamma _t$ for all $t\ge 1$ further implies

$$\begin{aligned} \ln \frac{W_n}{W_0}&\le \sum _{t=2}^n\frac{\gamma _{t-1}-\gamma _t}{\gamma _{t-1}}\ln \left|\mathcal {E}\right|-\gamma _n\sum _{t=1}^n\ell _P(h_{1:t})\\&=\sum _{t=2}^n\frac{\gamma _{t-1}-\gamma _t}{\gamma _{t-1}}\ln \left|\mathcal {E}\right|-\gamma _nL_P(h_{1:n})\\&\le \frac{1}{\gamma _n}\sum _{t=2}^n(\gamma _{t-1}-\gamma _t)\ln \left|\mathcal {E}\right|-\gamma _nL_P(h_{1:n})\\&=\frac{\gamma _1-\gamma _n}{\gamma _n}\ln \left|\mathcal {E}\right|-\gamma _nL_P(h_{1:n})\\&=\left( \frac{\gamma _1}{\gamma _n}-1\right) \ln \left|\mathcal {E}\right|-\gamma _nL_P(h_{1:n}). \end{aligned}$$

Replacing the definition of $\gamma _n$ in the first term, we get

$$\begin{aligned} \ln \frac{W_n}{W_0}\le (\root 4 \of {n}-1)\ln \left|\mathcal {E}\right|-\gamma _nL_P(h_{1:n}) \end{aligned}$$

(20)

Finally, putting together (19) and (20) yields

$$\begin{aligned} -\gamma _nL_\tau (h_{1:n})-\ln \left|\mathcal {E}\right|\le (\root 4 \of {n}-1)\ln \left|\mathcal {E}\right|-\gamma _nL_P(h_{1:n}) \end{aligned}$$

or, equivalently,

$$\begin{aligned} L_P(h_{1:n})-L_\tau (h_{1:n})\le \frac{\root 4 \of {n}}{\gamma _n}\ln \left|\mathcal {E}\right|=\sqrt{\frac{n}{2}\ln \left|\mathcal {E}\right|}. \end{aligned}$$

Because the above inequality holds for all $\tau$,

$$\begin{aligned} R_n(P,\mathcal {E})\le \sqrt{\frac{n}{2}\ln \left|\mathcal {E}\right|} \end{aligned}$$

$\square$

1.2 Proof of Theorem 2

Our proof involves three steps:

• In the first step, we show that, asymptotically, the ability of the ad hoc to predict future plays, given its initial belief over the target task, converges to that of an agent knowing the target task.

• In the second step, we derive an upper bound for the regret that depends linearly on the relative entropy between the probability distribution over trajectories conditioned on knowing the target task and the probability distribution over trajectories conditioned on the prior over tasks.

• Finally, in the third step, we use an information-theoretic argument to derive an upper bound to the aforementioned relative entropy, using the fact that the ad hoc agent is asymptotically able to predict future plays as well as if it knew the target task.

The proof is somewhat long and requires some notation and a number of auxiliary results, which are introduced in the continuation.

1.2.1 Preliminary results and notation

In this section, we introduce several preliminary results that will be of use in the proof. We start with a simple lemma due to Hoeffding [30] that provides a useful bound for the moment generating function of a bounded r.v. $X$.

Lemma 2

Let $X$ be a r.v. such that $\mathbb {E}_{}\left[ X\right] =0$ and $a\le X\le b$ almost surely. Then, for any $\lambda >0$,

$$\begin{aligned} \mathbb {E}_{}\left[ e^{-\lambda X}\right] \le e^{\frac{\lambda ^2(b-a)^2}{8}}. \end{aligned}$$

The following is a classical result from P. Levy [55, Theorem VII.3]. This result plays a fundamental role in establishing that the beliefs of the ad hoc agent asymptotically concentrate around the target task.

Lemma 3

Let $(\Omega , \mathcal {F}, \mathbb {P})$ be a probability space, and let $\left\{ \mathcal {F}_n,n=1,\ldots \right\}$ be a non-decreasing family of $\sigma$-algebras, $\mathcal {F}_1\subseteq \mathcal {F}_2\subseteq \ldots \subseteq \mathcal {F}$. Let $X$ be a r.v. with $\mathbb {E}_{}\left[ \left|X\right|\right] <\infty$ and let $\mathcal {F}_\infty$ denote the smallest $\sigma$-algebra such that $\bigcup _n\mathcal {F}_n\subseteq \mathcal {F}_\infty$. Then,

$$\begin{aligned} \mathbb {E}_{}\left[ X\mid \mathcal {F}_n\right] \rightarrow \mathbb {E}_{}\left[ X\mid \mathcal {F}_\infty \right] \end{aligned}$$

with probability 1.

We conclude this initial subsection by establishing several simple results. We write $\mathbb {E}_{p}\left[ \cdot \right]$ to denote the expectation with respect to the distribution $p$ and $\mathrm {D_{KL}}\left( p\Vert q\right)$ to denote the Kullback-Liebler divergence (or relative entropy) between distributions $p$ and $q$, defined as

$$\begin{aligned} \mathrm {D_{KL}}\left( p\Vert q\right) =\mathbb {E}_{p}\left[ \log \frac{p(X)}{q(X)}\right] . \end{aligned}$$

It is a well-known information-theoretic result (known as the Gibbs inequality) that $\mathrm {D_{KL}}\left( p\Vert q\right) \ge 0$.

Lemma 4

Let $X$ be a r.v. taking values in a discrete set $\mathcal {X}$ and let $p$ and $q$ denote two distributions over $\mathcal {X}$. Then, for any function ,

$$\begin{aligned} \mathbb {E}_{q}\left[ f(X)\right] \le \mathrm {D_{KL}}\left( q\Vert p\right) +\log \mathbb {E}_{p}\left[ e^{f(X)}\right] . \end{aligned}$$

Proof

Define, for $x\in \mathcal {X}$,

$$\begin{aligned} v(x)=\frac{e^{f(x)}p(x)}{\mathbb {E}_{p}\left[ e^{f(X)}\right] }. \end{aligned}$$

Clearly, $v(x)\ge 0$ and $\sum _xv(x)=1$, so $v$ is a distribution over $\mathcal {X}$. We have

$$\begin{aligned} \mathrm {D_{KL}}\left( q\Vert v\right)&=\mathbb {E}_{q}\left[ \log q(X)\right] -\mathbb {E}_{q}\left[ \log v(X)\right] \\&=\mathbb {E}_{q}\left[ \log q(X)\right] -\mathbb {E}_{q}\left[ f(X)\right] -\mathbb {E}_{q}\left[ \log p(X)\right] +\log \mathbb {E}_{p}\left[ e^{f(X)}\right] \\&=\mathrm {D_{KL}}\left( q\Vert p\right) -\mathbb {E}_{q}\left[ f(X)\right] +\ln \mathbb {E}_{p}\left[ e^{f(X)}\right] \end{aligned}$$

Because $\mathrm {D_{KL}}\left( q\Vert v\right) \ge 0$,

$$\begin{aligned} \mathrm {D_{KL}}\left( q\Vert p\right) -\mathbb {E}_{q}\left[ f(X)\right] +\log \mathbb {E}_{p}\left[ e^{f(X)}\right] \ge 0, \end{aligned}$$

and the result follows.$\square$

Lemma 5

(Adapted from [28]) Let $\left\{ a_n\right\}$ be a bounded sequence of non-negative numbers. Then, $\lim _na_n=0$ if and only if

$$\begin{aligned} \lim _n\frac{1}{n}\sum _{k=1}^na_k=0. \end{aligned}$$

Proof

Let $A=\sup _na_n$ and, given $\varepsilon >0$, define the set

By definition, $\lim _na_n=0$ if and only if $\left|M_\varepsilon \right|<\infty$ for any $\varepsilon$. Let $M_\varepsilon ^n=M_\varepsilon \cap \left\{ 1,\ldots ,n\right\}$. We have that

$$\begin{aligned} \frac{1}{n}\sum _{k=1}^na_n&=\frac{1}{n}\sum _{k\in M^n_\varepsilon }a_k+\frac{1}{n}\sum _{k\notin M^n_\varepsilon }a_k\\&\le \frac{A}{n}\left|M^n_\varepsilon \right|+\varepsilon . \end{aligned}$$

The conclusion follows.$\square$

1.2.2 Convergence of predictive distributions

We can now take the first step in establishing the result in Theorem 2. In particular, we show that, asymptotically, the ability of the ad hoc agent to predict future actions of the teammate player converges to that of an agent that knows the target task.

We start by noting that given the POMDP policy computed by Perseus and the initial belief for the ad hoc agent, it is possible to determine the probability of occurrence of any particular history of (joint) actions $h\in \mathcal {H}$. In particular, we write

$$\begin{aligned} \tilde{\mu }(h_{1:n})\triangleq \mathbb {P}_{}\left[ H(n)=h_{1:n}\mid p_0\right] \end{aligned}$$

and

$$\begin{aligned} \mu (h_{1:n})\triangleq \mathbb {P}_{}\left[ H(n)=h_{1:n}\mid \delta _{\tau ^*}\right] \end{aligned}$$

to denote the distribution over histories induced by the POMDP policy when the initial belief over tasks is, respectively, $p_0$ and $\delta _{\tau ^*}$. We write $\delta _{\tau }$ to denote the belief over tasks that is concentrated in a single task $\tau$. Note that, for a particular history $h_{1:n}=\left\{ a(1),\ldots ,a(n)\right\}$, we can write

$$\begin{aligned} \tilde{\mu }(h_{1:n})&=\prod _{t=1}^n\mathbb {P}_{}\left[ A(t)=a(t)\mid H(t-1)=h_{1:t-1},p_0\right] \\&=\prod _{t=1}^nP(H_N(t),B(h_{1:t-1},p_0)), \end{aligned}$$

where $B(h_{1:t-1},p_0)$ denotes the belief $p_{t-1}$ over tasks computed from the initial belief $p_0$ and the history $h_{1:t-1}$. Let $\mathcal {F}_n$ denote the $\sigma$-algebra generated by the history up to time-step $n$, i.e., the smallest $\sigma$-algebra that renders $H(n)$ measurable.¹⁸ The assertion we seek to establish is formalized as the following result.

Proposition 1

Suppose that $p_0(T^*)>0$, i.e., the initial belief of the agent over tasks does not exclude the target task. Then, with probability 1 (w.p.1), for every $\varepsilon >0$, there is $N>0$ such that, for every $m\ge n\ge N$,

$$\begin{aligned} 1-\varepsilon \le \frac{\mu (H(m)\mid \mathcal {F}_n)}{\tilde{\mu }(H(m)\mid \mathcal {F}_n)}\le 1+\varepsilon . \end{aligned}$$

Proof

Given a history $h'$ of length $n$, let $\mathcal {H}_{\mid h'}$ denote the subset of $\mathcal {H}$ that is compatible with $h'$, i.e., the set of histories $h\in \mathcal {H}$ such that $h_{1:n}=h'$. Our proof goes along the lines of [34]. We start by noticing that, because $p_0(T^*)>0$, it holds that

$$\begin{aligned} \mu (h)>0\Rightarrow \tilde{\mu }(h)>0, \end{aligned}$$

i.e., $\mu \ll \tilde{\mu }$ ($\mu$ is absolutely continuous w.r.t. $\tilde{\mu }$). We can thus define, for every finite history $h$, a function as

$$\begin{aligned} d(h)=\frac{\mu (h)}{\tilde{\mu }(h)}, \end{aligned}$$

with $d(h)=0$ if $\mu (h)=0$. From Lemma 3,

$$\begin{aligned} \mathbb {E}_{\tilde{\mu }}\left[ d(H)\mid \mathcal {F}_n\right] \rightarrow \mathbb {E}_{\tilde{\mu }}\left[ d(H)\mid \mathcal {F}_{\infty }\right] , \end{aligned}$$

(21)

which is an $\mathcal {F}_{\infty }$-measurable and strictly positive random variable. On the other hand, by definition, we have that

$$\begin{aligned} \mathbb {E}_{\tilde{\mu }}\left[ d(H)\mid \mathcal {F}_n\right]&=\mathbb {E}_{\tilde{\mu }}\left[ d(H)\mid H(n)\right] \\&=\sum _{h\in \mathcal {H}}d(h)\tilde{\mu }(h\mid H(n))\\&=\frac{1}{\tilde{\mu }(H(n))}\sum _{h\in \mathcal {H}_{\mid H(n)}}d(h)\tilde{\mu }(h)\\&=\frac{1}{\tilde{\mu }(H(n))}\sum _{h\in \mathcal {H}_{\mid H(n)}}\mu (h)\\&=\frac{\mu (H(n))}{\tilde{\mu }(H(n))}. \end{aligned}$$

We now note that, for $m\ge n, \mu (H(m)\mid H(n))$ is a r.v. such that

$$\begin{aligned} \mu (H(m)\mid H(n))= {\left\{ \begin{array}{ll} \frac{\mu (H(n))}{\mu (H(m))} &{} \text {if}\, H(m)\in {\mathcal {H}_m\mid H(n)}\\ 1 &{} \text {otherwise}, \end{array}\right. } \end{aligned}$$

(22)

with a similar relation holding for $\tilde{\mu }(H(m)\mid H(n))$. From (21), $\mathbb {E}_{\tilde{\mu }}\left[ d(H(m))\mid \mathcal {F}_n\right]$ converges w.p.1 to a strictly positive r.v. Then, for any $\varepsilon >0$, there is $N(\varepsilon )>0$ such that, for $m\ge n\ge N$,

$$\begin{aligned} 1-\varepsilon <\frac{\mathbb {E}_{\tilde{\mu }}\left[ d(H)\mid \mathcal {F}_m\right] }{\mathbb {E}_{\tilde{\mu }}\left[ d(H)\mid \mathcal {F}_n\right] }<1+\varepsilon . \end{aligned}$$

(23)

Combining (22) with (23), we finally get that, for any $\varepsilon >0$, there is $N(\varepsilon )>0$ such that, for $m\ge n\ge N$, w.p.1,

$$\begin{aligned} 1-\varepsilon <\frac{\mu (H(m)\mid H(n))}{\tilde{\mu }(H(m)\mid H(n))}<1+\varepsilon , \end{aligned}$$

and the proof is complete.$\square$

1.2.3 Initial upper bound

We now proceed by deriving an upper bound to the regret of any POMDP policy that depends on the relative entropy between the distributions $\mu$ and $\tilde{\mu }$, defined in ‘Convergence of predictive distributions section’ under Appendix 2. This bound will later be improved to finally yield the desired result.

Proposition 2

Suppose that $p_0(T^*)>0$, i.e., the initial belief of the agent over tasks does not exclude the target task. Then,

$$\begin{aligned} J_{\tau ^*}(n, \tau ^*)-J_P(n, \tau ^*)\le \mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) +\log \frac{n}{2} \end{aligned}$$

(24)

Proof

The expected reward received by following the POMDP policy from the initial belief $\delta _{\tau ^*}$ is given by

$$\begin{aligned} J_{\tau ^*}(n,\tau ^*)=\mathbb {E}_{\mu }\left[ \sum _{t=1}^nr(A(t))\mid \tau ^*\right] . \end{aligned}$$

Similarly, for the initial belief $p_0$, we have

$$\begin{aligned} J_P(n,\tau ^*)=\mathbb {E}_{\tilde{\mu }}\left[ \sum _{t=1}^nr(A(t))\mid \tau ^*\right] . \end{aligned}$$

Using Lemma 4,

$$\begin{aligned} \mathbb {E}_{\mu }\left[ \sum _{t=1}^nr(A(t))\mid \tau ^*\right] \le \mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) +\log \mathbb {E}_{\tilde{\mu }}\left[ e^{\sum _{t=1}^nr(A(t))}\mid \tau ^*\right] . \end{aligned}$$

Simple computations yield

$$\begin{aligned} \mathbb {E}_{\mu }\left[ \sum _{t=1}^nr(A(t))\mid \tau ^*\right]&\le \mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) +\log \mathbb {E}_{\tilde{\mu }}\left[ e^{\sum _{t=1}^nr(A(t))-\mathbb {E}_{\tilde{\mu }}\left[ r(A(t))\mid \tau ^*\right] }\mid \tau ^*\right] \\&\qquad +\mathbb {E}_{\tilde{\mu }}\left[ \sum _{t=1}^nr(A(t))\mid \tau ^*\right] . \end{aligned}$$

Because the exponential is a convex function, we can write

$$\begin{aligned} \mathbb {E}_{\mu }\left[ \sum _{t=1}^nr(A(t))\mid \tau ^*\right]&\le \mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) +\log \sum _{t=1}^n\mathbb {E}_{\tilde{\mu }}\left[ e^{r(A(t))-\mathbb {E}_{\tilde{\mu }}\left[ r(A(t))\mid \tau ^*\right] }\mid \tau ^*\right] \\&\qquad +\mathbb {E}_{\tilde{\mu }}\left[ \sum _{t=1}^nr(A(t))\mid \tau ^*\right] . \end{aligned}$$

Applying Lemma 2 to the second term on the right-hand side and replacing the definitions of $J_P$ and $J_{\tau ^*}$ yields

$$\begin{aligned} J_{\tau ^*}(n,\tau ^*)\le \mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) +\log \frac{n}{2}+J_P(n,\tau ^*), \end{aligned}$$

and the proof is complete.$\square$

1.2.4 Refined upper bound

We can now establish the result in Theorem 2. We use Lemma 5 and Proposition 1 to derive an upper bound for the term $\mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right)$, which appears in Proposition 2. The conclusion of Theorem 2 immediately follows.

Proposition 3

Suppose that $p_0(T^*)>0$, i.e., the initial belief of the agent over tasks does not exclude the target task. Then, the sequence $\left\{ d_n\right\}$ defined, for each $n$, as

$$\begin{aligned} d_n=\mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) \end{aligned}$$

is $o(n)$.

Proof

We start by noting that

$$\begin{aligned} \frac{\mu (H(n))}{\tilde{\mu }(H(n))}=\prod _{k=1}^n\frac{\mu (H(k)\mid H(k-1))}{\tilde{\mu }(H(k)\mid H(k-1)))}, \end{aligned}$$

which implies that

$$\begin{aligned} \mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) =\mathbb {E}_{\mu }\left[ \sum _{k=1}^n\log \frac{\mu (H(k)\mid H(k-1))}{\tilde{\mu }(H(k)\mid H(k-1)))}\right] . \end{aligned}$$

For each , define the sequences $\left\{ a_n\right\} , \left\{ b_n\right\}$ and $\left\{ c_n\right\}$ as

$$\begin{aligned} a_n&=\max \left\{ 0, \log \frac{\mu (H(k)\mid H(k-1))}{\tilde{\mu }(H(k)\mid H(k-1)))}\right\} , \\ b_n&=\max \left\{ 0, \log \frac{\tilde{\mu }(H(k)\mid H(k-1))}{\mu (H(k)\mid H(k-1)))}\right\} , \\ c_n&=a_n-b_n. \end{aligned}$$

By construction, we have $a_n>0$ and $b_n>0$ for all . Additionally, from Proposition 1,

$$\begin{aligned} \lim _na_n=\lim _nb_n=0 \end{aligned}$$

w.p.1, implying that $\lim _nc_n=0$ w.p.1. By Lemma 5, we have that, w.p.1,

$$\begin{aligned} \lim _n\frac{1}{n}\sum _{k=1}^nc_n=0 \end{aligned}$$

which finally yields that

$$\begin{aligned} \lim _n\frac{1}{n}\mathrm {D_{KL}}\left( \mu (H(n))\Vert \tilde{\mu }(H(n))\right) =0 \end{aligned}$$

and the proof is complete.$\square$