As an important task in remote sensing image processing, semantic segmentation of remote sensing images has broad application prospects in many fields such as disaster warning and rescue, environmental protection, and road planning. Research on semantic segmentation of remote sensing images based on deep learning has made some progress, but there are still problems such as poor perception of small object features, loss of detailed information in deep feature extraction, and imprecise segmentation contours of small objects. To this end, we propose a new remote sensing semantic segmentation model Swin-CDSA, which copes these problems to some extent by designing cascaded deep convolutional modules (CDCMs) and spatial attention mechanisms (SAMs). CDCM extracts multiscale features by using multilayer convolutions with different layers but parallel fixed small-sized kernels, while SAM supplements the model’s understanding of local and global information through a dual attention mechanism. We conducted experiments on the Potsdam and LoveDA datasets and achieved good results.