Abstract
I investigate relations between total performance of agent societies and relative performance of individual agents with respect to exploration ratio of multiagent learning. The exploration ratio is a key parameter to determine features of multiagent learning in two aspects: as a speed controller of learning in individual agents, and as a reciprocal noise factor for other agents. The investigation figures out trade-off of the two aspects and shows existence of single optimal value of the ratio to minimize the learning errors. I also carried out experiments to compare the performances of agents who use different exploration ratios. The results of the experiments tells existence of equilibrium points to choose the ratio by individual agents. Finally, we discuss the relationship between optimal and equilibrium values of the exploration ratio, which might bring dilemma of selection of the exploration ratio in an evolutionary way.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kaisers, M., Tuyls, K.: Frequency adjusted multi-agent q-learning. In: Proceedings of 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), pp. 309–315, May 2010
Noda, I.: Limitations of simultaneous multiagent learning in nonstationary environments. In: Proceedings of 2013 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT 2013), pp. 309–314. IEEE, November 2013
Noda, I.: Robustness of optimality of exploration ratio against agent population in multiagent learning for nonstationary environments. In: Multiagent Interaction Without Prior Coordination (Technical report WS-14-09), pp. 28–34. AAAI, July 2014
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Tokic, M.: Adaptive \(\epsilon \)-greedy exploration in reinforcement learning based on value differences. In: Dillmann, R., Beyerer, J., Hanebeck, U.D., Schultz, T. (eds.) KI 2010. LNCS, vol. 6359, pp. 203–210. Springer, Heidelberg (2010)
Tokic, M., Palm, G.: Value-difference based exploration: adaptive control between epsilon-greedy and softmax. In: Bach, J., Edelkamp, S. (eds.) KI 2011. LNCS, vol. 7006, pp. 335–346. Springer, Heidelberg (2011)
Wunder, M., Littman, M.L., Babes, M.: Classes of multiagent q-learning dynamicswith epsilon-greedy exploration. In: Frnkranz, J., Joachims, T. (eds.) Proceedings of the 27th International Conference on Machine Learning (ICML 2010), pp. 1167–1174. Omnipress (2010). http://www.icml2010.org/papers/191.pdf
Acknowledgments
This work was supported by JST CREST and JSPS KAKENHI 24300064.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A Unimodality of Lower Bound
A Unimodality of Lower Bound
Here, the lower bound is denoted as \(\mathcal{L}(\epsilon )\). Also, \(A = \frac{K \tilde{g}_{a}}{T}\) and \(B = \frac{K+1}{K}\) is introduced for the simplicity. So, \(\mathcal{L}\) can be defined as:
Because of the definition, \(1 < B \le 2\) is hold.
Lemma A1
\(\frac{\partial \mathcal{L}}{\partial \epsilon }\) is unimodal and convex upward. \(\square \)
Proof
Therefore, \(\frac{\partial \mathcal{L}}{\partial \epsilon }\) has only one extremal value, and is unimodal.
Therefore, \(\frac{\partial \mathcal{L}}{\partial \epsilon }\) is convex upward. \(\blacksquare \)
Lemma A2
Suppose that \(K \ge 3\). If the solution of \(\frac{\partial \mathcal{L}}{\partial \epsilon } = 0\) exists in the range \(0 < \epsilon < \frac{1}{2}\), the following inequality is hold:
\(\square \)
Proof
Suppose that the solution of \(\frac{\partial \mathcal{L}}{\partial \epsilon } = 0\) is \(\epsilon _0\):
Now, let’s consider the value of \(\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}}\):
Because \(K \ge 3\) and \(1 < B \le \frac{4}{3}\) are hold. So, we can get inequalitis, \( \epsilon _0^{-} \le 0 \) and \( \epsilon _0{+} \ge \frac{1}{2}\). Therefore, solutions of \(\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}} = 0\) exist one in range \(\le 0\) and two in range \(\ge \frac{1}{2}\).
Because \(\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}}\) is a cubic function of \(\epsilon _0\) and \(0 < \epsilon _0 < \frac{1}{2}\), \(\left. \frac{\partial \mathcal{L}}{\partial \epsilon }\right| _{\epsilon = \frac{1}{2}} > 0\) is hold. \(\blacksquare \)
Theorem A3
\(\mathcal{L}(\epsilon )\) is monotonically decreasing or is uni-modal and convex downward in the closed interval \([0,\frac{1}{2}]\). \(\square \)
Proof
Suppose that \(\frac{\partial \mathcal{L}}{\partial \epsilon } = 0\) has a solution in the closed interval \([0,\frac{1}{2}]\). Using Lemmas A1 and A2, we can say that the solution is only one in the interval.
Also, at the limit \(\epsilon \rightarrow 0\), \(\frac{\partial \mathcal{L}}{\partial \epsilon }\) is negative. Therefore, \(\mathcal{L}(\epsilon )\) is uni-modal and convex downward in the closed interval.
If \(\frac{\partial \mathcal{L}}{\partial \epsilon } = 0\) has no solution in the interval \([0,\frac{1}{2}]\), \(\mathcal{L}(\epsilon )\) is monotonically decreasing because \(\frac{\partial \mathcal{L}}{\partial \epsilon }\) is negative. \(\blacksquare \)
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Noda, I. (2016). Optimality and Equilibrium of Exploration Ratio for Multiagent Learning in Nonstationary Environments. In: Gaudou, B., Sichman, J. (eds) Multi-Agent Based Simulation XVI. MABS 2015. Lecture Notes in Computer Science(), vol 9568. Springer, Cham. https://doi.org/10.1007/978-3-319-31447-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-31447-1_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-31446-4
Online ISBN: 978-3-319-31447-1
eBook Packages: Computer ScienceComputer Science (R0)