Surface defect segmentation of no-service rail is important for its quality assessment. There are several challenges of uneven illumination, complex background, and difficulty of sample collection for no-service rail surface defects (NRSDs). In this article, we propose an acquisition scheme with two lamp light and color scan line charge-coupled device (CCD) to alleviate uneven illumination. Then, a multiple context information segmentation network is proposed to improve NRSD segmentation. The network makes full use of context information based on dense block, pyramid pooling module, and multi-information integration. Besides, the attention mechanism is applied to optimize extracted information by filtering noise. For the problem of real sample shortage, we propose to utilize artificial samples to train the network. And an NRSD data set NRSD-MN is built with artificial NRSDs and natural NRSDs. Experimental results show that our method is feasible and has a good segmentation effect on artificial and natural NRSDs.