Optimal action sequence generation for assistive agents in fixed horizon tasks

Abstract

Agents providing assistance to humans are faced with the challenge of automatically adjusting the level of assistance to ensure optimal performance. In this work, we argue that identifying the right level of assistance consists in balancing positive assistance outcomes and some (domain-dependent) measure of cost associated with assistive actions. Towards this goal, we contribute a general mathematical framework for structured tasks where an agent playing the role of a ‘provider’—e.g., therapist, teacher—assists a human ‘receiver’—e.g., patient, student. We specifically consider tasks where the provider agent needs to plan a sequence of actions over a fixed time horizon, where actions are organized along a hierarchy with increasing success probabilities, and some associated costs. The goal of the provider is to achieve a success with the lowest expected cost possible. We present OAssistMe, an algorithm that generates cost-optimal action sequences given the action parameters, and investigate several extensions of it, motivated by different potential application domains. We provide an analysis of the algorithms, including proofs for a number of properties of optimal solutions that, we show, align with typical human provider strategies. Finally, we instantiate our theoretical framework in the context of robot-assisted therapy tasks for children with Autism Spectrum Disorder (ASD). In this context, we present methods for determining action parameters based on a survey of domain experts and real child-robot interaction data. Our contributions unlock increased levels of flexibility for agents introduced in a variety of assistive contexts.

Appendix: Proofs

Our proofs are structured along the following three (mutually exclusive) cases:

• a. $O^*_1>0$, or equivalently $R<\min _a{c(a)/p(a)}$

• b. $O^*_1<0$, or equivalently $R>\min _a{c(a)/p(a)}$

• c. $O^*_1=0$, or equivalently $R=\min _a{c(a)/p(a)}$

Lemma 1

For any T, we have one of:

• a. $0<O^*_T<\min _a{c(a)/p(a)}-R$

• b. $0>O^*_T>\min _a{c(a)/p(a)}-R$

• c. $0=O^*_T=\min _a{c(a)/p(a)}-R$

Proof

We use induction on T. From Eq. (7): $O^*_T=\min _a{\{(1-p(a))O^{*}_{T-1}+c(a)-p(a)R\}}, \;\;\; O^*_1=\min _a{\{c(a)-p(a)R\}}$ applies in all cases.

Case (a):

Base case: $0<O^*_1=\min _a{c(a)-p(a)R}<\min _a{c(a)/p(a)}-R$

Induction step: Assume $0<O^*_{T-1}<\min _a{c(a)/p(a)}-R$, then $O^*_T$ is also positive from Eq. (7) and base case. Also, for all a:

$$\begin{aligned} \begin{aligned} O^*_T&\le (1-p(a))O^*_{T-1}+c(a)-p(a)R\\&<(1-p(a))(c(a)/p(a)-R)+c(a)-p(a)R =c(a)/p(a)-R \end{aligned} \end{aligned}$$

By induction, $0<O^*_T<\min _a{c(a)/p(a)}-R$ for all T.

Case (b):

Base case: $0>O^*_1=\min _a{c(a)-p(a)R}>\min _a{c(a)/p(a)}-R$

Induction step: Assume $0>O^*_{T-1}>\min _a{c(a)/p(a)}-R$. Also let $a^\dagger =\arg \min _a{c(a)/p(a)}$, let $a^*$ be the optimal action selected at stage T, and let $a^{*(1)}$ be the optimal action selected at stage 1.

$$\begin{aligned} O^*_T\le (1-p(a^{*(1)}))O^*_{T-1}+c(a^{*(1)})-p(a^{*(1)})R<0 \end{aligned}$$

since $(1-p(a))O^*_{T-1}<0$ for any a, and $c(a^{*(1)})-p(a^{*(1)})R<0$ (base case).

Also, for all a:

$$\begin{aligned} \begin{aligned} O^*_T&=(1-p(a^*))O^*_{T-1}+c(a^*)-p(a^*)R \\&>(1-p(a^*))(c(a^\dagger )/p(a^\dagger )-R)+c(a^*)-p(a^*)R \\&=(1-p(a^*))c(a^\dagger )/p(a^\dagger )+c(a^*)-R \end{aligned} \end{aligned}$$

Using $p(a^*)<p(a^\dagger )c(a^*)/c(a^\dagger )$:

$O^*_T>\left[ 1-p(a^\dagger )c(a^*)/c(a^\dagger )\right] c(a^\dagger )/p(a^\dagger )+c(a^*)-R =c(a^\dagger )/p(a^\dagger )-R$

By induction, $0>O^*_T>\min _a{c(a)/p(a)}-R$ for all T.

Case (c) is easily proven by induction on T. $\square$

Lemma 2

$O^*_T$ is monotonic in T. In particular, it is one of:

• a. Strictly increasing, i.e., $O^*_{T+1}>O^*_{T}$ for all T

• b. Strictly decreasing, i.e., $O^*_{T+1}<O^*_{T}$ for all T

• c. Constant, i.e., $O^*_{T+1}=O^*_{T}$ for all T

Proof

Let $a^*$ be the optimal action of stage T.

Case (a):

$O^*_T/O^*_{T-1}=1-p(a^*)+(c(a^*)-p(a^*)R)/O^*_{T-1}$

From Lemma 1, for any a: $0<O^*_{T-1}<c(a)/p(a)-R$, so $(c(a^*)-p(a^*)R)/O^*_{T-1}>p(a^*)$, hence: $O^*_T/O^*_{T-1}>1$, and $O^*_T>0$ for all T, which establishes that $O^*_T$ is strictly increasing.

Case (b):

The demonstration that $O^*_T/O^*_{T-1}>1$ is identical to case (a). Given that $O^*_T<0$ for all T, then $O^*_T$ is strictly decreasing.

Case (c) follows form the previous lemma.$\square$

Theorem 1

$O^*_T$ converges to $\min _a{c(a)/p(a)}-R$ as T goes to infinity.

Proof

Lemmas 1 and 2 imply convergence of $O^*_T$ in cases (a) and (b). Furthermore, setting $O_{T-1}$ to $O_T$ in Eq. (7) results in a single fixed point $\min _a{c(a)/p(a)}-R$, which establishes the result.

Case (c) is trivial since $\min _a{c(a)/p(a)}-R=0$.$\square$

Theorem 2

If $\varvec{\varPi }^*$ is an optimal sequence, then it is monotonic in t. In particular, $\varvec{\varPi }^*$ is one of:

• a. Nonincreasing, i.e., $a^*_1 \ge a^*_2 \ge \cdots \ge a^*_T$

• b. Nondecreasing, i.e., $a^*_1 \le a^*_2 \le \cdots \le a^*_T$

• c. Constant, i.e., $a^*_1= a^*_2=\cdots = a^*_T$

Proof

Let $a'$ be an optimal action associated with $O^*_{T-1}$ and $a''$ an optimal action associated with $O^*_T$. Then:

$$\begin{aligned}(1-p(a''))O^*_T+c(a'')-p(a'')R \le (1-p(a'))O^*_T+c(a')-p(a')R \\ (p(a')-p(a''))O^*_T\le c(a')-c(a'')-R(p(a')- p(a'')) \end{aligned}$$

(15)

and

$$\begin{aligned} (1-p(a'))O^*_{T-1}+c(a')-p(a')R \le (1-p(a''))O^*_{T-1}+c(a'')-p(a'')R \\ (p(a')-p(a''))O^*_{T-1}\ge c(a')-c(a'')-R(p(a')- p(a'')) \end{aligned}$$

(16)

Combining Eqs. (15) and (16), we get:

$$\begin{aligned} (p(a')-p(a''))O^*_{T}\le (p(a')-p(a''))O^*_{T-1} \end{aligned}$$

We can conclude that:

If $p(a')>p(a'')$: $O^*_T\le O^*_{T-1}$ and if $p(a')<p(a'')$: $O^*_T\ge O^*_{T-1}$

Assume $a'>a''$. Then $p(a')>p(a'')$. From the previous result, in case (a): $O^*_T\le O^*_{T-1}$, which contradicts Lemma 2. Hence, $a'\le a''$, which establishes that $\varvec{\varPi }^*$ is nonincreasing.

Similarly, we can show that, in case (b), $\varvec{\varPi }^*$ is nondecreasing.

In case (c), every step is equivalent to the single trial case, and the same action is selected at every trial, so the resulting sequence is constant.$\square$