The traditional convolutional neural networks (CNNs) coupled with cross-entropy loss ignore interclass relationship, and hence output unreasonable predictions from a holistic perspective. We address this issue by integrating CNNs with Wasserstein distance (WD): first, we find that the classical WD problem has an analytical solution in the case of multiclass classification; second, by leveraging multiple pretrained CNNs to extract multiscale convolutional features and encoding the features via the improved Fisher kernel, we propose a novel method for computing the ground distance matrix, which characterizes the affinities between classes and is also a key component of the WD problem; third, we use the analytical solution to construct new losses for CNNs. Our proposed model is applied to scene classification and leads to a higher performance than other methods.