Layered Controller Synthesis for Dynamic Multi-agent Systems

Abstract

In this paper we present a layered approach for multi-agent control problem, decomposed into three stages, each building upon the results of the previous one. First, a high-level plan for a coarse abstraction of the system is computed, relying on parametric timed automata augmented with stopwatches as they allow to efficiently model simplified dynamics of such systems. In the second stage, the high-level plan, based on SMT-formulation, mainly handles the combinatorial aspects of the problem, provides a more dynamically accurate solution. These stages are collectively referred to as the SWA-SMT solver. They are correct by construction but lack a crucial feature: they cannot be executed in real time. To overcome this, we use SWA-SMT solutions as the initial training dataset for our last stage, which aims at obtaining a neural network control policy. We use reinforcement learning to train the policy, and show that the initial dataset is crucial for the overall success of the method.

This work was partially funded by ANR project TickTac (ANR-18-CE40-0015).

Appendices

A Markov Decision Process for the Running Example

The Markov Decision Process is defined by its state space S, action space A, initial state distribution $p(s_0 \in S)$, reward function $r(s_t \in S, a_t \in A, s_{t+1} \in S)$ and deterministic transition function $s_{t+1} = \texttt {step}(s_t, a_t)$.

We describe here all the elements of the MDP defined for our running example:

• The state space ($\mathbb {R}^{720}$). The environment contains 3 paths, which are unions of sections, and as detailed in Sect. 2, we have imposed a maximum number of 3 cars per path, so there are at most 9 cars, to which we can attribute a unique identifier (we use $\{-1, 0, 1\}^2$). The state is entirely defined by the speed and position of each car. We could thus use vectors of size 18 to represent states, but instead we chose a sparser representation with a better structure. To remain coherent, we use the same road network presented in Sect. 2 which is composed of 24 section (since every car has a dedicated initial and goal node subdividing the sections containing initial and goal positions). On this road network, three different paths are defined, and each section being shared by at most 2 paths. At any given time any section may contain at most 6 cars by construction. For each section, we define a list of 6 tuples, all equal to (0, 0, (0, 0), 0) if no car is currently inside the section. However if there are cars in the section, say 2 cars for example, then the first two tuples have this structure:

  $$ (\texttt {position~with~the~section}, \texttt {normalized~velocity}, \texttt {car~identifier}, 1) $$

  We represent states as a concatenation of the values of all these tuples for all the 24 sections, which amounts to a vector of size 720. It is a sparse representation, but its advantage is that it makes it easy to find cars close to each other, as they are either in the same section or in neighbor sections.

• The action space ($\mathbb {R}^9$) and transition dynamics. Given an ordering of the 9 cars, an action is simply a vector of 9 accelerations. If $a_i$ is the acceleration for the car i, and if at the current time step its position within its path is $p_i$, and its speed is $v_i$, then at the next time step its position will be $p_i + v_i$, and its speed will be $v_i + a_i$. This defines the transition dynamics of the MDP. The components of an action corresponding to cars that are not present in the state are simply ignored. Remark: actions can be computed straightforwardly from a sequence of states as they are equal to the difference between consecutive speeds for each car.

• The reward. When all cars have reached their destination, i.e. crossed the end of their path, a reward of 2000 is given, and the episode is terminated. Besides, when there is either a collision (a violation of the safety distance between two cars) or two car facing each other in opposite directions in the same section, a negative reward (−100) is given and the episode is terminated. Finally, at each time step, two positive rewards are given, one proportional to the average velocity of the cars (to encourage cars to go fast), and one proportional to the (clamped) minimum distance between all cars (to encourage cars to stay far from each other). We set the maximum number of time step per episode to 85, and adjust these rewards so that an episode cannot reach a cumulated reward of 2000 unless it is truly successful and gets the final +2000 reward.

• The initial state distribution. We define an arbitrary initial state distribution in which each of the 9 cars has an 80% chance of being present. The speed of each car is defined randomly, and positions are also defined randomly (within roughly the first two third of each path). Safety distances are ensured, so that the inital states are not in collision, however speeds may be such that there will a collision after the first time step, so there is no guarantee of feasibility.

B Hyperparameters of the RL Algorithms

For TD3:

• Actor network architecture: multi-layer perceptron (MLP) [1] with 3 hidden layers of size 256 and rectified linear unit (ReLU) activation functions.

• Actor optimizer: ADAM [26], actor learning rate: $10^{-3}$

• Critic network architecture: MLP with 3 hidden layers of size 256 and ReLU activation functions.

• Critic optimizer: ADAM, critic learning rate: $10^{-3}$

• Discount factor: 0.99

• Soft update coefficient ($\tau$): 0.05

For TD3BC:

• Actor network architecture: MLP with 3 hidden layers of size 256 and ReLU activation functions.

• Actor optimizer: ADAM, actor learning rate: $10^{-3}$

• Critic network architecture: MLP with 3 hidden layers of size 256 and ReLU activation functions.

• Critic optimizer: ADAM, critic learning rate: $10^{-3}$

• Discount factor: 0.99

• Soft update coefficient ($\tau$): 0.05

• $\alpha$: 2.5