Speckle removal is a crucial preliminary step for synthetic aperture radar (SAR) image processing. In recent years, the application of deep neural networks toward solving SAR image despeckling problems has yielded commendable outcomes. However, prevailing deep learning methods for SAR image despeckling rely on convolutional neural network (CNN) architectures, which inherently capture only local information within their receptive fields. Consequently, the despeckling performance can be further improved through advanced network structure design. More recently, the transformers-based methods show impressive performance in natural image denoising relying on the long-range dependency modeling capability of the self-attention mechanism. In this letter, we introduce a practical despeckling network that incorporates the global modeling capability of the Swin transformers (SwinTs) and the local modeling ability of the residual CNNs. Specifically, the proposed despeckling network is based on the widely used U-Net architecture, wherein a Swin Conv (SC) block is adopted to replace the convolutional layer in the baseline UNet. The SC block mainly comprises a residual convolutional (RConv) block and a SwinT block, which are used to extract local features and long-range dependencies from images, respectively. Furthermore, to deal with the spatially correlated real SAR speckle, a pixel-shuffle downsampling (PD) post-processing strategy is adopted, which can significantly improve the practicality of the proposed method without additional real dataset for fine-tuning. Experimental results demonstrate that the proposed method achieves the state-of-the-art performance on both synthetic and real SAR images, and outperforms CNN-based and transformer-based methods ( $L=1$ ) by an average peak signal-to-noise ratio (PSNR) of 0.94 and 1.58 dB, respectively.