Facial expression recognition (FER) is a key factor in human behavior analysis. Most algorithms deal with FER as a pure classification problem, assuming that expressions are exclusive to each other. In this letter, the problem of FER is tackled from a more detailed view: learning to discriminate expressions with consideration of the secondary information. We propose the Secondary Information aware Facial Expression Network (SIFE-Net) to explore the latent components without auxiliary labeling, and we propose a novel dynamic weighting strategy to teach the SIFE-Net. In contrast to traditional classifiers trained with one-hot labels, the proposed SIFE-Net takes advantage of secondary expression information and has more rational feature distributions. We carry out extensive experiments and analysis on three widely-used FER datasets, i.e. the CK+ dataset, the JAFFE dataset, and the RAF dataset. Experimental results show that the SIFE-Net achieves state-of-the-art performance on all three datasets, which demonstrates the effectiveness of our method.