Safety and security are the most concerned aspects in railway level crossing. Existing method are mainly focus on danger evaluation or obstacle detection of level crossing traffic. This research work present a railway level crossing control system based on a combination of video analysis and position methods to deal with the problems: autonomous control of level crossing gate and over-the-horizon visual perception for train driver. We proposed a video analysis method based on object detection and prediction network to make up the open-gate decision the trackside subsystem. To increase the detection accuracy, an improved YOLOv5 network based on difference feature fusion module (DFFM) is proposed. Furthermore, a prediction network (TPD-net) using 1D convolution is presented to predict whether train has left, therefore the gate of level crossing can be opened without risk. Experiments show that proposed DFFM is able to improve detection performance, and TPD-net works accurately and stably in train passage detection. Our system can be efficient and safe for autonomous control in railway level crossing system.