As the research on deep learning methods gradually progresses, more and more classification models are applied in the classification of hyperspectral image (HSI). High-dimensional and low-resolution characteristics of HSI, however, make it difficult for conventional models to process its data effectively. In this article, a novel HSI classification model, namely, spatial–spectral pyramid network (SSPN), is designed by combining a 3-D convolutional neural network (3D CNN) with feature pyramid structure. SSPN taking advantage of 3-D convolution coupled with multiscale convolutional extraction is used to obtain a large set of diverse spatial–spectral features. Multiscale interfusion is also applied in SSPN to enrich the features contained in a single feature map and to improve the sensitivity on HSI spatial–spectral information, allowing it to better learn spatial–spectral features. Moreover, the losses of each combination based on multiscale interfusion are calculated via weighted average, which enables SSPN to avoid the excessive influence of single combination in the updating of model parameters. Four HSI public datasets and several comparison models are employed to validate the classification effect of SSPN. Experimental results show that SSPN achieves the highest overall accuracy (OA) in all datasets compared with other classification models, with 100%, 98.8%, 99.8%, and 98.7% on the datasets of Chikusei, Pavia University, Botswana, and Houston 2013, respectively. SSPN is demonstrated to possess higher classification accuracy and better generalization performance on HSI.