Automatic modulation recognition (AMR) plays an important role in modern wireless communication. In this letter, a novel framework for AMR is proposed. The ResNeXt network serves as the backbone, and four proposed adaptive attention mechanism modules are incorporated. The time-frequency representations of the received signals are utilized as the inputs of the proposed deep learning (DL) network, and a transfer learning strategy is adopted based on the pre-trained ResNeXt weakly supervised learning (WSL) model. The comparisons with several state-of-the-art techniques on the RadioML2016.10B and RadioML2018.01A datasets show that the proposed framework converges quickly and can achieve higher robustness and 2% to 8% higher accuracy than other state-of-the-art techniques.