Depression has become one of the serious mental health diseases in the world. The computer vision-based methods are expected to assist the clinical diagnosis of depression more efficiently and objectively, but the lack of clinical data and the low accuracy of recognition have hindered the broad application of automatic depression diagnosis. Given the shortcomings in predicting depression, this paper proposed a deep network based on the deep residual regression network to predict the severity of depression from facial expressions, named Deep Residual Regression Convolutional Neural Networks (DRR_DepressionNet). We firstly enhanced the original facial images to expanse the training data. Then we use these training data, which carry different feature information, to train a deep regression residual network (ResNet). Unlike the traditional ResNet network, we divided the network into three major modules, namely C_M block, Resblock, and GAP. We also replaced the cross-entropy loss function in the traditional structure by the Euclidean loss function as the basis for training the network. Finally, we applied the trained network to predict the Beck Depression Inventory (BDI) score of new subjects to reflect the severity of depression. The experiments were validated on AVEC2013 and AVEC2014 depression data, respectively. The experimental results showed that compared with the state-of-the-art performance, the proposed method could improve the RMSE and MAE by 2.4% and 0.3% respectively on the AVCE2013 data set, and improve the RMSE and MAE by 0.3% and 1.1% respectively on AVCE2014 data set.