Existing deepfake detection methods have achieved high accuracy under the intra-database scenario, but most of them suffer from a significant performance drop when evaluated on cross-database experiments. In this paper, we regard the deepfake detection task as domain generalization problem and design a Facial Decomposition-based Domain Generalization framework to learn a more generalized feature representation. To optimize our framework, we experimentally analysis that current CNN-based detectors tend to overfit the facial semantic content but neglect the common shared traces of deepfake, thus a facial semantic content decomposition based dual-branch network is designed to reduce the representation discrepancies among multiple source domains in a specific feature space to be domain invariant. Besides, based on the consideration that the distribution discrepancies are much larger among the fake faces than the real ones, a hybrid loss function is designed to enlarge the distance among samples from different categories even in cross-domain scene. Extensive experiments demonstrate that the proposed method can achieve better performance compared with state-of-the-art methods, especially on cross-database evaluation.