Packet routing in communication networks is a natural problem for sequential decision-making. Previously, many heuristic methods are proposed based on domain knowledge. Most of them rely on the understanding of the network environment and the algorithms are sensitive to the model of the networks. However, in practice it is intractable to model the dynamics of a real network perfectly, since many features of the network such as traffic load and network topology are irregularly changing with time and the changes are hard to predict. Hence, model-free approach becomes more promising in dealing with a complex dynamic environment. In this work, we consider a model-free method by leveraging reinforcement learning techniques. We propose a multi-agent reinforcement learning framework for adaptive routing in communication networks, which takes advantage of both the real-time Q-learning and the actor-critic methods. Provided a global feedback signal, the routers (agents) act independently but are able to learn cooperative behaviors to reduce packet delivery time. Our algorithm is robust to some dynamic changes in the network and each agent learns an adaptive policy to route packets. Simulation results demonstrate that our proposed algorithm outperforms some existing benchmark algorithms.