Railway intrusion seriously threatens railway safety and can cause enormous loss of life and property. Therefore, railway intrusion detection is crucial for the safety of railway operation. Among the current methods of intrusion detection, machine vision-based methods have been widely used in railways, and have attracted close attention because of their great benefits. This paper proposes a comprehensive review of railway intrusion detection based on machine vision, covering ground monitoring, on-board inspection, and unmanned aerial vehicle (UAV) inspection. First, this paper systematically reviews most of the studies over the past two decades and presents the survey in three parts. Second, by analyzing these studies and the requirements for railway monitoring, we summarize the major challenges that hinder railway intrusion detection based on machine vision. Finally, we propose several promising perspectives for railway intrusion detection based on machine vision by comprehensively considering the development of machine vision, sensors, and pattern recognition together with the needs of railway scenes.