Convolutional Neural Network (CNN) achieves impressive success in image super-resolution (SR), where global context interaction is critical for reconstructing reliable edge and texture details. However, most CNN-based SR models focus on modeling the global contextual information within a single feature map by using attention mechanisms, and ignore the dependencies among hierarchical features, resulting in blurred or even distorted detail restoration, especially for SR tasks with large scaling factors (i.e., $\times 4$ , $\times 8$ ). To tackle the above issue, we propose a Cross-Layer interaction pyramid Super-Resolution (CLSR) network that reconstructs the desired SR images progressively in a coarse-to-fine fashion. Specifically, we propose a novel Cross-Layer Non-Local attention (CLNL) for accurate detail restoration. Through explicitly modeling the long-range feature-wise similarities within and between layers, the proposed CLNL is able to discriminatively explore complementary patches from hierarchical features to reconstruct the target LR patches. Then, to further strengthen the information interaction among hierarchical features at different scales, we propose a novel Gradient Consistency-Aware learning framework (GCA) by constructing a closed loop (LR $\rightarrow$ HR $\rightarrow$ LR) on the gradient space. The proposed GCA is able to effectively capture the interdependence between LR and HR gradient maps to guide our CLSR for reliable detail restoration. Extensive experiments validate that our CLSR outperforms the state-of-the-art methods in terms of both reconstruction accuracy and visual quality.