Generative Adversarial Network (GAN) has been employed for single image super-resolution (SISR). However, unregularized GAN is difficult for training. This is due gradient descent based GAN optimization is not easy to convergence, thus limiting its performance for image super-resolution. In this paper, a relativistic super-resolution GAN with a simplified gradient penalty (RSRGAN-GP) is proposed for single image super-resolution. In the proposed method, a compact residual network optimized by removing Batch-Normalization layers is employed as the generator to estimate photo-realistic images of 4× upscaling. Further, we introduce a residual network, which also has no Batch-Normalization layers as the conditional discriminator and adopt a simplified gradient regularization to penalize it for stabilizing the super-resolution GAN training, thus guaranteeing high-quality image reconstruction. Additionally, the super-resolution GAN is enhanced with a relativistic discriminator, which produces sharp and rich-detail images at no extra computational cost. The results on benchmark datasets show that our proposed method can effectively improve the visual quality of super-resolved images and achieves competitive performance compared with related works.