A Cooperation Online Reinforcement Learning Approach in Ant-Q

Abstract

Online reinforcement learning achieves learning after update estimation value for (state, action) pairs selecting in present state before do state transition by next state. Therefore, online reinforcement learning needs polynomial search time to find most optimal value-function. But, a lots of reinforcement learning that are proposed for online reinforcement learning update estimation value for (state, action) pairs that agents select in present state, and because estimation value for unselected (state, action) pairs is evaluated in other episodes, perfect online reinforcement learning is not. Therefore, in this paper, we propose online ant reinforcement learning method using Ant-Q and eligibility trace to solve this problem. The eligibility trace is one of the basic mechanisms in reinforcement learning to handle delayed reward. The traces are said to indicate the degree to which each state is eligible for undergoing learning changes should a reinforcing event occur. Formally, there are two kinds of eligibility traces(accumulating trace or replacing traces). In this paper, we propose online ant reinforcement learning algorithms using an eligibility traces which is called replace-trace methods. This method is a hybrid of Ant-Q and eligibility traces. Although replacing traces are only slightly different from accumulating traces, it can produce a significant improvement in optimization. We could know through an experiment that proposed reinforcement learning method converges faster to optimal solution than Ant Colony System and Ant-Q.