The latest developments in single image super-resolution (SISR) combine the power of convolutional neural networks (CNN) to produce better performance. However, as the depth and width of the networks increase, CNN-based super-resolution methods have faced the challenges of computational complexity in practice. Feature redundancy in images is an essential characteristic of efficient CNNs, but it is rarely studied in the design of super-resolution methods. In this paper, we remove the redundancy of features in the space and channel dimensions, propose a fast and accurate SISR model called the spatial and channel feature distillation network (SCFDN). Specifically, SCFDN first compresses and splits the features along the space and channel dimensions to skip redundant calculations, then further processes the detailed features, and finally fuses all the features to obtain the HR image. We propose an inverse sub-pixel convolutional layer (ISPC) to map features from the spatial dimension to the channel dimension, so that the effect of feature distillation is improved. Our experimental results confirm that the proposed SCFDN has a better trade-off compared to the state-of-the-art methods in terms of performance and runtime. The source code is available at https://github.com/cyfwry/SCFDN-pytorch.