Virtual Markets: Q-Learning Sellers with Simple State Representation

Abstract

As shopbots spread through Internet collecting information about lowest prices / highest qualities human sellers will turn out to be too slow to tune the prices and thus unprofitable in comparison with smart agents — pricebots. One of the most promising approaches to building pricebots is Q-learning. Its advantages: flexibility to act under changing conditions of virtual markets, Q-learning sellers can take into account not only immediate rewards but also profits far ahead, and don’t need information neither on buyer demand nor on competitors’ behaviour. But up to now Q-learning sellers used state representation exponential in the number of sellers acting in the market and could function successfully only with one competitor which no doubt is unrealistic. We are proposing a new state representation independent of the number of sellers that allowed to 10 agents to find the prices that maximize cumulative profits under conditions of high competition in three moderately realistic economic models. It was also shown that due to their flexibility Q-learning sellers managed to collect more profit than pricebots based on two other generally used approaches even though one of them possessed much more information about buyer demand and competitors’ behaviour. The proposed representation doesn’t depend on the number of sellers and in principle Q-learning sellers using it can function in the markets with arbitrary number of competitors.