This paper introduces a novel way to improve the settling time of transitions between different walking controllers. This improvement is achieved by commanding a sequence of intermediate transitions to the target controller. As a result, the state of the system enters the domain of attraction of the target controller closer to the fixed point of the Poincaré Map. The method is applicable to any walking robot with one degree of underactuation. The problem is expressed as a Markov Decision Process and then solved with Reinforcement Learning. In order to simplify the stability analysis of underactuated walking the Hybrid Zero Dynamics framework is utilized. Another advantage of using the Hybrid Zero Dynamics is the dimensionality reduction of the state representation in the Markov Decision Process. The experimental results suggest that the proposed methodology performs better than a one-step transition for 84.34% of all the considered transitions for a simulated walking robot matching the parameters of RABBIT [1].