In recent years, deep convolutional neural networks (CNNs) have been widely investigated for hyperspectral image (HSI) classification. The CNN-based HSI classifiers obtained good performance under the condition of sufficient training samples. In order to address the problem of limited training samples, in this letter, transfer learning is combined with CNN to address the issue of HSI classification. Pretrained models on large-scale data sets (e.g., ImageNet) can extract the general and discriminative features. Due to the fact that the extracted low-level and mid-level features can be reused for the HSI feature extraction, the CNN-based methods usually obtain good classification performance with insufficient training samples. The ImageNet data set has three channels, while the HSI data set contains hundreds of channels. Therefore, three channels of the HSI are randomly selected to formulate a transferring CNN. Then, several transferring CNNs are combined to establish an ensemble classification system with diversity. Moreover, an improved label smoothing technique is proposed to further improve the classification accuracy of the HSI. Experimental results on two popular hyperspectral data sets [i.e., Indian Pines and Kennedy Space Center (KSC)] show that the transferring CNN ensemble obtains good classification performance compared to the state-of-the-art methods.