Transfer-based attacks against black-box neural network models have received increasing attention because they are more realistic scenarios, but how to produce highly transferable adversarial examples on the surrogate model becomes critical. In this work, we find that if the attack direction of the original example is controlled from the beginning, the produced adversarial examples will be more transferable. Specifically, we propose the Output Direction Controller (ODC) to initialize the example direction so that the example starts off with a deviation from the true direction or toward the target direction. ODC is a simple and extensible component that can be combined with various transfer-based attack methods and significantly improve the transferability of the adversarial examples. On the ImageNet dataset, we optimize the baseline method by ODC to improve the success rate of untargeted attacks by an average of 11.79% and targeted attacks by an average of 3.38%. Code is available at https://github.com/yangrongbo/ODC.