Imbalanced datasets are deathful for model training. In the field of steering prediction, imbalanced training is the core reason that model is unable to predict sharp steering value well. This paper proposes a new loss framework to train the robust end-to-end model, which is named SteeringLoss. The imbalanced distribution of steering value for datasets is analyzed, which is similar with Gaussian distribution. With the feature of distribution, the gain factor (1+α|y|^β)^γ is added to square loss function. This new SteeringLoss framework is able to improve the impact of sharp steering value while maintain the impact of small steering value. Experiment results show the SteeringLoss based model performs higher performance than traditional square loss based model with higher prediction precision and wider prediction range. Meanwhile, γ is able to control the time of training process, and β can control the model performance, different distribution needs to choose different pair of parameters. What's more, the SteeringLoss framework is suitable for imbalanced training with similar distribution of dataset. The code can be found at: https://github.com/weiy1991/SteeringLoss.