In transformer for semantic segmentation, patch embedding usually has only one convolutional layer with a large stride, leading to the decrease of feature extraction capability. In addition, the complex decoder results in high computation cost. To address the abovementioned two issues, we put forward a progressive downsampling transformer with convolution-based decoder (PDCDT), which is a simple, efficient yet powerful framework. Specifically, progressive downsampling layers for patch embedding are designed to refine the extracted features and reduce information loss at each stage of the hierarchical transformer encoder. Meanwhile, a simple decoder based on a convolution (conv) module is proposed for aggregating the characteristic information from multiscale (MS) output layers of the encoder, and it can realize dimensional transformation and information interaction with fewer parameters than the decoders used in the existing transformers. Extensive experiments show that PDCDT achieves competitive results on ADE20K mean intersection over union (47.9% mIoU) and cityscapes (82.6% mIoU). Finally, PDCDT is applied to gear pitting measurement in the gear contact fatigue test, and the comparative results indicate that PDCDT can improve the accuracy of pitting detection.