Solving uncapacitated P-Median problem with reinforcement learning assisted by graph attention networks

Abstract

The P-Median Problem is one of the basic cases of facility location problems and has been studied for many years. Most methods of solving it are based on classical heuristics or meta-heuristic and they don’t perform well on large-scale problems according to time cost. In this paper, we propose the first reinforcement learning-based method which uses Multi-Talking-Heads Graph Attention Networks to learn representations and design a learnable attention mechanism to solve the uncapacitated P-Median Problem. We train the model using REINFORCE algorithm and show that it has good performance on uncapacitated P-Median Problem according to solution quality and time consumption. We also apply our model to the realistic dataset and empirically figure out that the difference between data distributions is one of the most important factors to influence the final performances.

Appendix : A: Implementation Details of Transformer Module

Attention mechanism is embedded in the Transformer [3]. In this section we present some technical details about the implementation. For a node within a graph, the attention means the weights of messages should receive from its neighbor nodes. This node’s weight of messages depends on the compatibility between its query and the key of its neighbors. Formally, we denote the node representation as $h_{i}\in \mathbb {R}^{d_{h}}$ and matrices $W^{K},W^{Q}\in \mathbb {R}^{d_{k}\times d_{h}},W^{V}\in \mathbb {R}^{d_{v}\times d_{h}}$ and compute the key $k_{i}\in \mathbb {R}^{d_{k}}$, value $v_{i}\in \mathbb {R}^{d_{v}}$, query $q_{i}\in \mathbb {R}^{d_{k}}$ for each node in the following way:

$$ q_{i} = W^{Q} h_{i},\ k_{i} = W^{K} h_{i},\ v_{i} = W^{V} h_{i}, $$

(21)

From these queries and keys, we can obtain the compatibility of node i’s query and node j’s key: $u_{ij}\in \mathbb {R}$ as the scaled dot-product [3]:

$$ u_{ij} = \begin{cases} \frac{{q_{i}}^{T} {k}_{j}}{\sqrt{d_{\text{k}}}} & \text{if } \text{node } i \text{ is adjacent to } j\\ -\infty & \text{otherwise.} \end{cases} $$

(22)

We then compute the attention weights a_ij ∈ [0, 1] using softmax function:

$$ a_{ij}=\frac{e^{u_{ij}}}{{\sum}_{j^{\prime}}e^{u_{ij^{\prime}}}} $$

(23)

At last, we complete the information aggregation to get a new node representation by:

$$ h_{i}^{\prime}={\sum}_{j} a_{ij}v_{j} $$

(24)

Furthermore, as noted in [3], it’s beneficial to introduce Multi-head Attention which has multiple attention heads which can help to capture different kinds of information from different neighbors. In particular, we parameterize a set of matrices $\{{W^{K}_{m}},{W^{Q}_{m}}\in \mathbb {R}^{d_{k}\times d_{h}},{W^{V}_{m}}\in \mathbb {R}^{d_{v}\times d_{h}},m=1,2,...M\}$ and obtain M node representations for node i by above attention mechanism $\{ h_{i,m}^{\prime },m=1,2,...M\}$. The final multi-head attention value for node i is obtained by:

$$ MHA_{i}(h_{1},...,h_{n})=\sum\limits_{m=1}^{M}{W_{m}^{O}}h_{i,m}^{\prime} $$

(25)

where $\{{W_{m}^{O}}\in \mathbb {R}^{d_{h}\times d_{v}},m=1,2,...M\}$

At the implementation level, we need some extra neural networks modules to obtain these representations: Feed-forward sublayer and batch normalization [48].

The feed-forward sublayer computes each node’s representations using a linear projection followed a ReLU activation:

$$ FF(h_{i})=W^{ff,1}\cdot ReLU(W^{ff,0}h_{i}+b^{ff,0})+b^{ff,1} $$

(26)

The batch normalization aims to reduce internal covariate shift during training neural networks and the formula with a batch of inputs {x^{(k),k= 1,2,...}} is as follows:

$$ \hat{x}^{(k)}=\frac{x^{(k)}-E[x^{(k)}]}{\sqrt{Var[x^{(k)}]}} $$

The overall pipeline of this process is shown in Fig. 10

Fig. 10

The pipeline of Transformer module