Routing is critical for maximizing the performance of Dragonfly networks. It decides how the packets are forwarded from their source nodes to their destination nodes. Considering the traffic pattern varies over time in real networks, adaptive routing is desirable. Existing adaptive routing algorithms employ local information to make dynamic routing decisions, which have shown significant limitations since the local information typically fails to reflect the global network condition. In this paper, inspired by the hierarchical topology of Dragonfly, we develop a hierarchical Reinforcement Learning (RL) algorithm named Q-hierarchical for Dragonfly networks. Q-hierarchical learns to adapt from data without the need to model the traffic pattern. It simplifies the complexity of traditional RL-based routing by routing in a hierarchical manner, i.e., inter-group routing and intra-group routing. Hence it can be applied on a large network. We also develop a fast and effective learning strategy for the hierarchical RL. The performance of Q-hierarchical is evaluated through comprehensive tests on two Dragonfly topologies. The results show that our approach provides comparable performance under uniform random traffic pattern and outperforms some routing algorithms in terms of averaging packet delay and affordable load under adversarial traffic pattern.