In this paper, we introduce a new approach to detect synthetic portrait images and videos. Motivated by the observation that the symmetry of synthetic facial area would be easily broken, this approach aims to reveal the tampering trace by features learned from symmetrical facial regions. To do so, a two-stream learning framework is designed which uses a hard sharing Deep Residual Networks as the backbone network. The feature extractor maps the pair of symmetrical face patches to an angular distance indicating the difference of symmetry features. Extensive experiments are carried out to test the effectiveness in detecting synthetic portrait images and videos, and corresponding results show that our approach is effective even on heterogeneous data and re-compression data that were not used to train the detection model.