Concurrent imitation dynamics in congestion games

Abstract

Imitating successful behavior is a natural and frequently applied approach when facing complex decision problems. In this paper, we design protocols for distributed latency minimization in atomic congestion games based on imitation. We propose to study concurrent dynamics that emerge when each agent samples another agent and possibly imitates this agent’s strategy if the anticipated latency gain is sufficiently large. Our focus is on convergence properties. We show convergence in a monotonic fashion to stable states, in which none of the agents can improve their latency by imitating others. As our main result, we show rapid convergence to approximate equilibria, in which only a small fraction of agents sustains a latency significantly above or below average. Imitation dynamics behave like an FPTAS, and the convergence time depends only logarithmically on the number of agents. Imitation processes cannot discover unused strategies, and strategies may become extinct with non-zero probability. For singleton games we show that the probability of this event occurring is negligible. Additionally, we prove that the social cost of a stable state reached by our dynamics is not much worse than an optimal state in singleton games with linear latency functions. We concentrate on the case of symmetric network congestion games, but our results do not use the network structure and continue to hold accordingly for general symmetric games. They even apply to asymmetric games when agents sample within the set of agents with the same strategy space. Finally, we discuss how the protocol can be extended such that, in the long run, dynamics converge to a pure Nash equilibrium.

Appendix

Throughout the technical part of this paper, we apply the following two Chernoff bounds.

Fact 7

(Chernoff, see [24]) Let $X$ be a sum of Bernoulli variables. Then, $\mathbb {P}_{}\left[ X \ge k\cdot \mathbb {E}\left[ X\right] \right] \le \mathrm {e}^{-\mathbb {E}\left[ X\right] \,k\cdot (\ln k - 1)}$, and, for $k\ge 4 > \mathrm {e}^{4/3}, \mathbb {P}_{}\left[ X \ge k\cdot \mathbb {E}\left[ X\right] \right] \le \mathrm {e}^{-\frac{1}{4}\,\mathbb {E}\left[ X\right] \,k\,\ln k}$. Equivalently, for $k\ge 4\,\mathbb {E}\left[ X\right] , \mathbb {P}_{}\left[ X \ge k\right] \le \mathrm {e}^{-\frac{1}{4}\,k\,\ln (k/\mathbb {E}\left[ X\right] )}$.

The following fact yields a linear approximation of the exponential function.

Fact 8

For any $r>0$ and $x\in [0,r]$, it holds that $(\mathrm {e}^{x} - 1) \le x\cdot \frac{\mathrm {e}^{r}-1}{r}$.

Proof

The function $\exp (x)-1$ is convex and it goes through the points $(0,0)$ and $(r,\mathrm {e}^{r}-1)$, as does the function $x\cdot \frac{\mathrm {e}^{r}-1}{r}$. $\square$

Fact 9

It holds that

$$\begin{aligned} \sum _{k=1}^{\infty } \mathrm {e}^{-k(\ln k)}\cdot k < 2. \end{aligned}$$

Proof

We have

$$\begin{aligned} \sum _{k=1}^{\infty } \mathrm {e}^{-k(\ln k)}\cdot k \!=\! \sum _{k=1}^{\infty } \frac{1}{k^{k-1}} \!= \! 1 \!+\! \sum _{k=2}^{\infty } \frac{1}{k^{k-1}} \! < \! 1 \!+\! \sum _{k=1}^{\infty } \frac{1}{2^k} \! \le 2. \end{aligned}$$

$\square$

Fact 10

It holds that

$$\begin{aligned} \sum _{k=2}^{\infty } \mathrm {e}^{-k(\ln (k)-1)}\cdot k < 8. \end{aligned}$$

Proof

We have

$$\begin{aligned}&\sum _{k=2}^{\infty } \mathrm {e}^{-k(\ln (k)-1)}\cdot k = \sum _{k=1}^{\infty } \mathrm {e}^{} \cdot \left( \frac{\mathrm {e}^{}}{k+1}\right) ^k \, = \, \sum _{k=1}^{4} \mathrm {e}^{} \cdot \left( \frac{\mathrm {e}^{}}{k+1}\right) ^k \\&\qquad \qquad \quad \quad + \sum _{k=5}^{\infty } \mathrm {e}^{} \cdot \left( \frac{\mathrm {e}^{}}{k+1}\right) ^k < 7.1 + \mathrm {e}^{} \cdot \sum _{k=5}^{\infty } \frac{1}{2^k}\, < \, 8. \end{aligned}$$

$\square$

Fact 11

For every $c \in ]0,1[$ it holds

$$\begin{aligned} \sum _{k=0}^{\infty } c^k&= \frac{c}{1-c} \\ \sum _{k=l}^{\infty } c^k&= \frac{c^l}{1-c} \\ \end{aligned}$$

Fact 12

(Jensen’s Inequality) Let $f :\mathbb {R}\rightarrow \mathbb {R}$ be a convex function, and let $a_1,\ldots ,a_k,x_1,\ldots ,x_k \in \mathbb {R}$. Then

$$\begin{aligned} f \left( \frac{\sum _{i=1}^k a_i x_i}{\sum _{i=1}^k a_i} \right) \le \frac{\sum _{i=1}^k a_i f(x_i)}{\sum _{i=1}^k a_i}. \end{aligned}$$

If $f(x) = x^2$, then

$$\begin{aligned}&\left( \frac{\sum _{i=1}^k a_i x_i}{\sum _{i=1}^k a_i} \right) ^2 \le \frac{\sum _{i=1}^k a_i (x_i)^2}{\sum _{i=1}^k a_i} \\&\quad \Leftrightarrow \frac{1}{\sum _{i=1}^k a_i} \cdot \left( \sum _{i=1}^k a_i x_i \right) ^2 \le \sum _{i=1}^k a_i f(x_i). \end{aligned}$$

Lemma 7

Let $X_0,X_1,\ldots$ denote a sequence of non-negative random variables and assume that for all $i\ge 0$

$$\begin{aligned} \mathbb {E}\left[ X_i \mid X_{i-1}=x_{i-1}\right] \le x_{i-1} - 1 \end{aligned}$$

and let $\tau$ denote the first time $t$ such that $X_t=0$. Then,

$$\begin{aligned} \mathbb {E}\left[ \tau \mid X_0=x_0\right] \le x_0. \end{aligned}$$

The proof follows, e.g., from standard martingale arguments in combination with the optional stopping theorem and is omitted here.

Lemma 8

Let $X_0,X_1,\ldots$ denote a sequence of non-negative random variables and assume that for all $i\ge 0\mathbb {E}\left[ X_i \mid X_{i-1} = x_{i-1}\right] \le x_{i-1} \cdot \alpha$ for some constant $\alpha \in (0,1)$. Furthermore, fix some constant $x^*\in (0,x_0]$ and let $\tau$ be the random variable that describes the smallest $t$ such that $X_t \le x^*$. Then,

$$\begin{aligned} \mathbb {E}\left[ \tau \mid X_0=x_0\right] \le \frac{4}{1-\alpha } \cdot \ln \left( \frac{2x_0}{x^*}\right) . \end{aligned}$$

Proof

Let us define $\gamma = \frac{1}{1-\alpha }$ and an auxiliary random variable $Y^t$ by $Y^0:= X^{0}$, and for any round $t \ge 1$,

$$\begin{aligned} Y^{t}&= \left\{ \begin{array}{ll} X^{t} &{} \mathrm{if} X^t > x^* \\ 0 &{} \mathrm{otherwise}. \end{array}\right. \end{aligned}$$

Then, for any $t \ge 1$, it follows

$$\begin{aligned} \mathbb {E}\left[ Y^t \mid X^{t-1}=x \right] \le \alpha x. \end{aligned}$$

We have for $\kappa = \gamma \cdot (\ln (x^0) - \ln (x^*/2))$,

$$\begin{aligned} \mathbb {E}\left[ Y^t \right]&= \sum _x \mathbb {E}\left[ Y^{t} \mid X^{t-1}=x \right] \cdot \mathbb {P}_{}\left[ X^{t-1}=x \right] \\&\le \sum _x \alpha \cdot Y^{t-1} \cdot \mathbb {P}_{}\left[ X^{t-1}=x \right] \\&\le \alpha ^{\tau } \cdot Y^{0} \le x^*/2. \end{aligned}$$

Hence by Markov’s inequality,

$$\begin{aligned} \mathbb {P}_{}\left[ Y^{\kappa } \ge x^* \right] \le \frac{1}{2}. \end{aligned}$$

(9)

We consider two cases.

Case 1: For all time steps $t \in [0,\ldots ,\kappa ], Y^{t} = X^{t}$. Then, as seen above $X^{\kappa } \le x^*$ with probability at least 1/2.

Case 2: There exists a step $t \in [1,\ldots ,\kappa ]$ such that $Y^{t} \ne X^{t}$. Let $t$ be the smallest time step with that property. Hence, $Y^{t} \ne X^{t}$, but $Y^{t-1} = X^{t-1}$. If $Y^{t-1}=0$, then $X^{t-1} = 0$. If $Y^{t-1} \ne 0$, then by definition of $Y^{t}$,

$$\begin{aligned} \left( Y^{t} \ne X^{t} \right) \bigwedge \left( Y^{t-1} \ne 0 \right) \Rightarrow X^t \le x^*. \end{aligned}$$

In all cases we have shown that with probability at least 1/2, there exists a step $t \in [0,\kappa ]$ so that $X^{t} \le x^*$. If such a step does not exist, we simply repeat the analysis and consider the next $\kappa$ steps. The probability that we do not observe a step as desired decreases exponentially in the number of restarts. In expectation, we need only $\sum _{k=1}^{\infty } k/2^{k-1} = 4$ phases of $\kappa$ steps to observe a step as desired. Thus, the expected number of steps is at most $\tau = 4\kappa$. This completes the proof of the lemma. $\square$