Low Signal-to-Noise Ratio (SNR) conditions pose significant challenges in Automatic Modulation Recognition (AMR) tasks. In this letter, we propose an innovative Multi-Dimensional Shrinkage Block (MDSB) to address these challenges. MDSB is a novel Convolutional Neural Network (CNN) architecture that effectively enhances the noise robustness of CNNs by employing a unique denoising mechanism, which tackles the limitations of CNNs in extracting temporal information. Leveraging the MDSB, a new AMR network named the Spatial and Channel-wise Shrinkage Neural Network (SCSNN) is introduced. Comprehensive experiments on multiple public datasets demonstrate the superior recognition performance of the proposed SCSNN model in comparison to other methods.