Existing video rain streaks removal methods utilize various manual models to represent the appearance of rain streaks, and only use convolutional neural network (CNN) as a post-processing part to compensate the artifacts like misalignment caused by traditional de-raining operations. However, these manual models only work for some particular scenes because the distribution of rain streaks is complex and random. Moreover, since CNN network and previous traditional de-raining operations cannot be trained jointly, the output of CNN network may still contain artifacts. To address these problems, we propose an end-to-end video rain streaks removal CNN network called EEVRSR net. Experimental results of both synthetic and real data demonstrate that the proposed EEVRSR net achieves better performance in both speed and effectiveness over state-of-the-art methods.