Facial expression recognition (FER) plays a vital role in affective cognition. However, there will be some limitations when facing the FER with single facial image data. Considering that extra data contains more information for molding, the facial action unit (AU) can be adopted as privileged information (PI) to assist the FER task. This article integrates AU information into an end-to-end deep network to support FER training. The proposed privileged action unit network (PAU-Net) gives ways of integrating AU information from the input aspect (type I) and output aspect (type II). Type I of PAU-Net takes AUs as input to guide the facial image network learning, which provides the AU-based emotion recognition result for the image-based FER model. While, type II of PAU-Net utilizes AUs as the output label for shallow layers of the network, which helps the model learn AU- related features and further assists advanced facial expression feature learning in subsequent layers. Note that PI enhances the network during the training and will not occur during the testing. Therefore, the network can still perform robustly with original input data in practice. Experiments are based on the CK+, MMI, and Oulu-CASIA data sets. The experimental results demonstrate the effectiveness of the proposed PAU-Net in FER tasks.