In signal compression, distortion information is significant for rate distortion optimization. In this paper, we propose a convolutional neural network (CNN) to predict distortion information for H.265/HEVC. With the strong representation power of CNN, structural similarity (SSIM) maps indicating distortion information can be predicted directly in an end-to-end, pixel-to-pixel way. Different from traditional CNNs which focus on learning one-to-one mappings from input to output, we show that our CNN model can predict SSIM maps conditioned on quantization parameters (QPs), realizing one-to-many mappings. To construct our CNN network, QP labels are designed as conditions to feed the CNN model. We also apply symmetrical network architecture and multi-level feature fusion method to ensure our network can utilize both high-level semantic features and low-level structure features. The experiments on MS COCO database demonstrate the effectiveness of our CNN-based method for SSIM prediction.