Simulation environments play a centric role in the research of sensor fusion and robot control. This paper presents Pavilion, a novel open-source simulation system, for robot perception and kinematic control based on the Unreal Engine and the Robot Operating System (ROS). The novelty of this work includes threefold: (1) developing a shader-based method to generate optical flow ground-truth data with the Unreal Engine, (2) developing a toolset to remove binary incompatibility between ROS and the Unreal Engine to enable real-time interaction, and (3) developing a method to directly import Simulation Description Format (SDF) robot models into the Unreal Engine at runtime. Finally, a Gazebo-compatible real-time simulation system is developed to enable training and evaluation of a large number of sensor fusion, planning, decision and control algorithms. The system can be implemented on both Linux and macOS, with the latest version of ROS. Various experiments have been performed to validate the superior performance of the proposed simulation environment over other state-of-the-art simulators in terms of number of modalities, simulation accuracy, latency and degree of integration difficulty.