Optimality and Equilibrium of Exploration Ratio for Multiagent Learning in Nonstationary Environments

Abstract

I investigate relations between total performance of agent societies and relative performance of individual agents with respect to exploration ratio of multiagent learning. The exploration ratio is a key parameter to determine features of multiagent learning in two aspects: as a speed controller of learning in individual agents, and as a reciprocal noise factor for other agents. The investigation figures out trade-off of the two aspects and shows existence of single optimal value of the ratio to minimize the learning errors. I also carried out experiments to compare the performances of agents who use different exploration ratios. The results of the experiments tells existence of equilibrium points to choose the ratio by individual agents. Finally, we discuss the relationship between optimal and equilibrium values of the exploration ratio, which might bring dilemma of selection of the exploration ratio in an evolutionary way.

A Unimodality of Lower Bound

Here, the lower bound is denoted as $\mathcal{L}(\epsilon )$. Also, $A = \frac{K \tilde{g}_{a}}{T}$ and $B = \frac{K+1}{K}$ is introduced for the simplicity. So, $\mathcal{L}$ can be defined as:

$$\begin{aligned} \mathcal{L}(\epsilon )= & {} T\sigma ^2 + A \epsilon ^{-1} + \epsilon N (2 - B \epsilon ) \end{aligned}$$

Because of the definition, $1 < B \le 2$ is hold.

Lemma A1

$\frac{\partial \mathcal{L}}{\partial \epsilon }$ is unimodal and convex upward.    $\square$

Proof

$$\begin{aligned} \frac{\partial ^2 \mathcal{L}}{\partial \epsilon ^2} = 2 A \epsilon ^{-3} - 2 N B = 0,&\quad&\therefore \epsilon = \root 3 \of {\frac{A}{NB}} \end{aligned}$$

Therefore, $\frac{\partial \mathcal{L}}{\partial \epsilon }$ has only one extremal value, and is unimodal.

$$\begin{aligned} \frac{\partial ^3 \mathcal{L}}{\partial \epsilon ^3}= & {} - 6 A \epsilon ^{-4} < 0 \end{aligned}$$

Therefore, $\frac{\partial \mathcal{L}}{\partial \epsilon }$ is convex upward.    $\blacksquare$

Lemma A2

Suppose that $K \ge 3$. If the solution of $\frac{\partial \mathcal{L}}{\partial \epsilon } = 0$ exists in the range $0 < \epsilon < \frac{1}{2}$, the following inequality is hold:

$$\begin{aligned} \left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}}> & {} 0 . \end{aligned}$$

   $\square$

Proof

Suppose that the solution of $\frac{\partial \mathcal{L}}{\partial \epsilon } = 0$ is $\epsilon _0$:

$$\begin{aligned} \left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \epsilon _0} = -A \epsilon _0^{-2} + 2 N - 2 N B \epsilon _0 = 0 ,&\quad&\therefore A = 2 N \epsilon _0^2 (1 - B \epsilon _0) . \end{aligned}$$

Now, let’s consider the value of $\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}}$:

$$\begin{aligned} \left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}}= & {} N (\epsilon - \frac{1}{2}) (\epsilon - \epsilon _0^{+}) (\epsilon - \epsilon _0^{-}) \\ \epsilon _0^{\pm }= & {} \frac{-(B-2) \pm \sqrt{-3 (B-2) (B+\frac{2}{3})}}{4 B} . \end{aligned}$$

Because $K \ge 3$ and $1 < B \le \frac{4}{3}$ are hold. So, we can get inequalitis, $\epsilon _0^{-} \le 0$ and $\epsilon _0{+} \ge \frac{1}{2}$. Therefore, solutions of $\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}} = 0$ exist one in range $\le 0$ and two in range $\ge \frac{1}{2}$.

Because $\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}}$ is a cubic function of $\epsilon _0$ and $0 < \epsilon _0 < \frac{1}{2}$, $\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}} > 0$ is hold.    $\blacksquare$

Theorem A3

$\mathcal{L}(\epsilon )$ is monotonically decreasing or is uni-modal and convex downward in the closed interval $[0,\frac{1}{2}]$.    $\square$

Proof

Suppose that $\frac{\partial \mathcal{L}}{\partial \epsilon } = 0$ has a solution in the closed interval $[0,\frac{1}{2}]$. Using Lemmas A1 and A2, we can say that the solution is only one in the interval.

Also, at the limit $\epsilon \rightarrow 0$, $\frac{\partial \mathcal{L}}{\partial \epsilon }$ is negative. Therefore, $\mathcal{L}(\epsilon )$ is uni-modal and convex downward in the closed interval.

If $\frac{\partial \mathcal{L}}{\partial \epsilon } = 0$ has no solution in the interval $[0,\frac{1}{2}]$, $\mathcal{L}(\epsilon )$ is monotonically decreasing because $\frac{\partial \mathcal{L}}{\partial \epsilon }$ is negative.    $\blacksquare$