The precondition that most of the existing facial expression recognition (FER) algorithms have succeeded lies in that the training (source) and test (target) samples are independent of each other and identically distributed. However, it is too strict to satisfy this precondition in the real-world. To this end, we propose a novel graph-diffusion-based domain-invariant representation learning (GDRL) model for the cross-domain FER scenario where there exist distribution shifts between various domains. Specifically, a low-dimensional space mapping strategy is first adopted to diminish the domain mismatch. Then, by skillfully combining the local graph embedding and affinity graph diffusion, the local geometric structures can be effectively modeled and the deeper higher-order relationships of samples from various domains can be captured. In addition, in order to better guide the transfer process and learn a more discriminative and invariant representation, we take into account the label consistency. Experimental results on four laboratory-controlled databases and two in-the-wild databases demonstrate that our proposed model can yield better recognition performance compared with state-of-the-art domain adaptation methods.