Recently, convolutional neural networks (CNNs) have been widely used for image processing problems. In this work, we present an end-to-end depth map super-resolution method based on CNN. Standing on a residual learning architecture, the proposed network learns joint features to get a high-resolution (HR) depth map from a low-resolution (LR) one with the multi-layers guidance of a HR color image. Furthermore, in order to focus on the boundaries of depth map, we generate an edge-attention map from the associated HR color images as a guidance. Experimental results show that the proposed network outperforms the state-of-the-art depth map super-resolution methods.