Hierarchical reinforcement learning via dynamic subspace search for multi-agent planning

Abstract

We consider scenarios where a swarm of unmanned vehicles (UxVs) seek to satisfy a number of diverse, spatially distributed objectives. The UxVs strive to determine an efficient plan to service the objectives while operating in a coordinated fashion. We focus on developing autonomous high-level planning, where low-level controls are leveraged from previous work in distributed motion, target tracking, localization, and communication. We rely on the use of state and action abstractions in a Markov decision processes framework to introduce a hierarchical algorithm, Dynamic Domain Reduction for Multi-Agent Planning, that enables multi-agent planning for large multi-objective environments. Our analysis establishes the correctness of our search procedure within specific subsets of the environments, termed ‘sub-environment’ and characterizes the algorithm performance with respect to the optimal trajectories in single-agent and sequential multi-agent deployment scenarios using tools from submodularity. Simulated results show significant improvement over using a standard Monte Carlo tree search in an environment with large state and action spaces.

Appendices

Submodularity

We review here concepts of submodularity and monotonicity of set functions following Clark et al. (2016). A power set function $f:2^\varOmega \rightarrow \mathbb {R}$ is submodular if it satisfies the property of diminishing returns,

$$\begin{aligned} f(X\cup \{x\}) - f(X) \ge f(Y\cup \{x\})-f(Y), \end{aligned}$$

(A.1)

for all $X\subseteq Y\subseteq \varOmega$ and $x\in \varOmega \setminus Y$. The set function f is monotone if

$$\begin{aligned} f(X) \le f(Y), \end{aligned}$$

(A.2)

for all $X\subseteq Y \subseteq \varOmega$. In general, monotonicity of a set function does not imply submodularity, and vice versa. These properties play a key role in determining near-optimal solutions to the cardinality-constrained submodular maximization problem defined by

$$\begin{aligned} \begin{aligned}&\max \,\,f(X)\\&\text {s.t. } |X| \le k. \end{aligned} \end{aligned}$$

(A.3)

In general, this problem is NP-hard. Greedy algorithms seek to find a suboptimal solution to (A.3) by building a set X one element at a time, starting with $|X|=0$ to $|X|=k$. These algorithms proceed by choosing the best next element,

$$\begin{aligned} \underset{x\in \varOmega \setminus X}{\max } f(X\cup \{x\}), \end{aligned}$$

to include in X. The following result (Clark et al. 2016; Nemhauser et al. 1978) provides a lower bound on the performance of greedy algorithms.

Theorem A.1

Let $X^*$ denote the optimal solution of problem (A.3). If f is monotone, submodular, and satisfies $f(\emptyset )=0$, then the set X returned by the greedy algorithm satisfies

$$\begin{aligned} f(X)\ge (1-e^{-1})f(X^*). \end{aligned}$$

An important extension of this result characterizes the performance of a greedy algorithm where, at each step, one chooses an element x that satisfies

$$\begin{aligned} f(X\cup \{x\})-f(X)\ge \alpha (f(X\cup \{x^*\})-f(X)), \end{aligned}$$

for some $\alpha \in [0,1]$. That is, the algorithm chooses an element that is at least an $\alpha$-fraction of the local optimal element choice, $x^*$. In this case, the following result (Goundan and Schulz 2007) characterizes the performance.

Theorem A.2

Let $X^*$ denote the optimal solution of problem (A.3). If f is monotone, submodular, and satisfies $f(\emptyset )=0$, then the set X returned by a greedy algorithm that chooses elements of at least $\alpha$-fraction of the local optimal element choice satisfies

$$\begin{aligned} f(X)\ge (1-e^{-\alpha })f(X^*). \end{aligned}$$

A generalization of the notion of submodular set function is given by the submodularity ratio (Das and Kempe 2011), which measures how far the function is from being submodular. This ratio is defined as largest scalar $\lambda \in [0,1]$ such that

$$\begin{aligned} \lambda \le \frac{\sum \limits _{z\in Z} f(X \cup \{z\}) -f(X)}{f(X\cup Z)-f(X)}, \end{aligned}$$

(A.4)

for all $X,Z\subset \varOmega$. The function f is called weakly submodular if it has a submodularity ratio in (0, 1]. If a function f is submodular, then its submodularity ratio is 1. The following result (Das and Kempe 2011) generalizes Theorem A.1 to monotone set functions with submodularity ratio $\lambda$.

Theorem A.3

Let $X^*$ denote the optimal solution of problem (A.3). If f is monotone, weakly submodular with submodularity ratio $\lambda \in (0,1]$, and satisfies $f(\emptyset )=0$, then the set X returned by the greedy algorithm satisfies

$$\begin{aligned} f(X)\ge (1-e^{-\lambda })f(X^*). \end{aligned}$$

Scenario optimization

Scenario optimization aims to determine robust solutions for practical problems with unknown parameters (Ben-Tal and Nemirovski 1998; Ghaoui et al. 1998) by hedging against uncertainty. Consider the following robust convex optimization problem defined by

$$\begin{aligned} \begin{aligned} \text {RCP: }&\min \limits _{\gamma \in \mathbb {R}^d }{c^T\gamma } \\&\text {subject to: } f_\delta (\gamma )\le 0,\,\, \forall \delta \in \varDelta , \end{aligned} \end{aligned}$$

(B.1)

where $f_{\delta }$ is a convex function, d is the dimension of the optimization variable, $\delta$ is an uncertain parameter, and $\varDelta$ is the set of all possible parameter values. In practice, solving the optimization (B.1) can be difficult depending on the cardinality of $\varDelta$. One approach to this problem is to solve (B.1) with sampled constraint parameters from $\varDelta$. This approach views the uncertainty of situations in the robust convex optimization problem through a probability distribution $Pr^{\delta }$ of $\varDelta$, which encodes either the likelihood or importance of situations occurring through the constraint parameters. To alleviate the computational load, one selects a finite number $N_{\text {SCP}}$ of parameter values in $\varDelta$ sampled according to $Pr^{\delta }$ and solves the scenario convex program (Campi et al. 2009) defined by

$$\begin{aligned} \begin{aligned} \text {SCP}_N :&\underset{\gamma \in \mathbb {R}^d}{\min }\,\,c^T\gamma \\&\text {s.t. } f_{\delta ^{(i)}}(\gamma )\le 0,\,\, i=1,\ldots ,N_{\text {SCP}}. \end{aligned} \end{aligned}$$

(B.2)

The following result states to what extent the solution of (B.2) solves the original robust optimization problem.

Theorem B.1

Let $\gamma ^*$ be the optimal solution to the scenario convex program (B.2) when $N_{\text {SCP}}$ is the number of convex constraints. Given a ‘violation parameter’, $\varepsilon$, and a ‘confidence parameter’, $\varpi$, if

$$\begin{aligned} N_{\text {SCP}}\ge \frac{2}{\varepsilon }\left( \text {ln}\frac{1}{\varpi }+d\right) \end{aligned}$$

then, with probability $1-\varpi$, $\gamma ^*$ satisfies all but an $\varepsilon$-fraction of constraints in $\varDelta$.

List of symbols

$({\mathbb {Z}}_{\ge 1}){\mathbb {Z}}$ :

(Non-negative) integers

$({\mathbb {R}}_{>0})\mathbb {R}$ :

(Positive) real numbers

|Y|:

Cardinality of set Y

$s\in S$ :

State/state space

$a\in A$ :

Action/action space

$Pr^s$ :

Transition function

r, R:

Reward, reward function

$(\pi ^*)\pi$ :

(Optimal) policy

$V^{\pi }$ :

Value of a state given policy, $\pi$

$\gamma$ :

Discount factor

$\alpha ,\beta$ :

Agent indices

${\mathcal {A}}$ :

Set of agents

$o\in {\mathcal {O}}^{b}\in {\mathcal {O}}$ :

Waypoint/objective/objective set

${\mathcal {E}}$ :

Environment

$x\in \varOmega _x$ :

Region/region set

${\mathcal {O}}^{b}_{i}$ :

Set of waypoints of an objective in a region

$s^{b}_{i}$ :

Abstracted objective state

$s_{i}$ :

Regional state

$\tau$ :

Task

$\varGamma$ :

Set of feasible tasks

$\overrightarrow{x}$ :

Ordered list of regions

$\xi _{\overrightarrow{x}}$ :

Repeated region list

$\phi _t(,)$ :

Time abstraction function

$(\epsilon _k)\epsilon$ :

(Partial) sub-environment

$N_{\epsilon }$ :

Size of a sub-environment

$\overrightarrow{\tau }$ :

Ordered list of tasks

$\overrightarrow{\text {Pr}^t}$ :

Ordered list of probability distributions

$(\vartheta ^p_{\beta })\vartheta$ :

(Partial) task trajectory

${\mathcal {I}}_{\alpha }$ :

Interaction set of agent $\alpha$

${\mathcal {N}}$ :

Max size of interaction set

$\theta$ :

Claimed regional objective

$\varTheta _{\alpha }$ :

Claimed objective set of agent $\alpha$

$\varTheta ^{{\mathcal {A}}}$ :

Global claimed objective set

$\varXi$ :

Interaction matrix

Q :

Value for choosing $\tau$ given $\epsilon . s$

${\hat{Q}}$ :

Estimated value for choosing $\tau$ given $\epsilon . s$

N :

Number of simulations in $\epsilon . s$

$N_{{\mathcal {O}}^{b}}$ :

Number of simulations of an objective in $\epsilon . s$

t :

Time

T :

Multi-agent expected discounted reward per task

$\lambda$ :

Submodularity ratio

${\hat{\lambda }}$ :

Approximate submodularity ratio

$f_{x}$ :

Sub-environment search value set function

$X_{x}\subset Y_{x}\subset \varOmega _x$ :

Finite set of regions

$f_{\vartheta }$ :

Sequential multi-agent deployment value set function

$X_{\vartheta }\subset Y_{\vartheta }\subset \varOmega _{\vartheta }$ :

Finite set of trajectories

$\varpi$ :

Confidence parameter

$\varepsilon$ :

Violation parameter