Although deep face recognition has achieved impressive results in recent years, controversy has arisen regarding racial and gender bias of the models, questioning their deployment into sensitive scenarios. This work quantifies for the first time the demographic imbalance of popular public face datasets in terms of identity, gender and ethnicity. We also publicly release DemogPairs, a new validation set with 10.8K facial images and 58.3M identity verification pairs, distributed in demographically-balanced folds of Asian, Black and White females and males. A benchmark of experiments is carried out using DemogPairs over state-of-the-art deep face recognition models (SphereFace, FaceNet and ResNet50), in order to analyze their cross-demographic behavior. Experimental results demonstrate that studied models suffer from a very structured and damaging demographic bias. Our experiments shine a light on novel testing protocols to appropriately validate the generalization capabilities of face recognition models.