The Convolution Neural Network (CNN) has achieved great performance in image classification, partially due to the deeper and deeper structure. While its complexity brings more challenge to the practical application. So we can design a more efficient loss function to get better results without a more complex network. In this paper, we proposed a weighted softmax loss function called confusion weighted loss to learn the relationship among the confusing categories. Firstly, we generate a similarity matrix based on the confusion matrix to illustrate the relationship among the categories. Then, we propose a clustering algorithm to find out the confusing categories. Finally, to learn more information among them, we design a weighted matrix used in our loss function. Experiments on different datasets demonstrate the effectiveness of our method.