Although color images are easily interfered by glass, depth images captured by infrared sensors are robust to reflection. In this letter, we propose multi-modal reflection removal using convolutional neural networks (CNNs). We build a multi-modal CNN for reflection removal to separate transmission from reflection using depth information. The proposed network consists of two sub-networks: image restoration and depth adaptation. Image restoration sub-network (IRN) recovers transmission layer from the input image with reflection, whereas depth adaptation sub-network (DAN) guides reflection removal of the IRN. Moreover, to extract image details for reflection removal, we present a multi-scale loss function that penalizes non-similarity for multi-scale outputs. Experimental results demonstrate that the proposed method is robust to dominant reflections and outperforms state-of-the-art methods in terms of both peak signal-to-noise ratio (PSNR) and structural similarity.