Global contextual information and spatial structural details are pivotal elements in the context of super-resolution (SR) reconstruction for remote sensing images. Therefore how to generate rich contextual semantic information and accurate spatial structure information simultaneously is a key challenge for remote sensing image SR. In this paper, we propose a novel progressive multi-scale learning strategy based on residual prior to solve the remote sensing image SR problem. In particular, we propose a novel progressive up-down mapping unit (PUMU) that asymptotically maps the input low-dimensional vectors into a high-dimensional space to learn global context information, which avoids loss of global information. Subsequently, we suggest introducing a novel method of explicitly mining spatial structure information, called residual prior (RP), which can help the proposed model to achieve spatial-structure-preserving SR. We have conducted extensive experiments on two public datasets including UCMerced and PatternNet, and the experimental results demonstrate the effectiveness of the proposed method.