It is of great significance to design a stable and reliable routing protocol for Vehicular Ad Hoc Networks (VANETs) that adopt Vehicle to Vehicle (V2V) communications in the face of frequent network topology changes. In this paper, we propose a hybrid routing algorithm, RCRIQ, based on improved Q-learning. For an established cluster structure, the cluster head is used to select the gateway vehicle according to the gateway utility function to expand the communication range of the cluster further. During the link construction stage, an improved Q-learning algorithm is adopted. The corresponding neighbor vehicle is chosen according to the maximum Q value in the neighbor list. The heuristic algorithm selects the next-hop by the maximum heuristic function value when selecting the next-hop neighbor node. The above two strategies are comprehensively evaluated to determine the next hop. This way ensures the optimal selection of the next hop in terms of reachability and other communication parameters. Simulation experiments show that the algorithm proposed in this article has better performance in terms of routing stability, throughput, and communication delay in the urban traffic scene.