Scalloping is one of the critical problems in ScanSAR images. It not only affects image visualization, but also influences the quantitative applications such as surface wind and wave retrievals in the ocean area. The existing method of descalloping needs artificial parameter setting and lacks generality in the image domain. A novel deep neural network based on residual learning for descalloping of ScanSAR images is proposed in this paper. The proposed method can eliminate scalloping patterns and has strong adaptive ability, which can handle inhomogeneous scalloping patterns and different scenarios. Experiments on GF-3 ScanSAR images verify the good performance of this method. The code for our models is available online.