Fine-Grained Image Classification is an inherently challenging task because of its inter-class similarity and intra-class variance. Most existing studies solve this problem by localization-and-classification strategies, which, however, always causes the problem of information loss or heavy computational expenses. Instead of localization-and-classification strategy, we propose a novel end-to-end optimization procedure named Multi-Task Attention Learning (MTAL), which reinforces the neural network’ correspondence to attention regions. Experimental results on CUB-Birds and Stanford Cars show that our procedure distinctly outperforms the baselines and is comparable with state-of-the-art studies despite its simplicity*.