3D object detection is a critical task in the field of robotics and autonomous driving. Although recent light detection and ranging (LiDAR)-based 3D object detection techniques have been well-studied and achieve high detection accuracy, the cost of the LiDAR sensor itself proposes a high premium for its practical implementation. Recently introduced Pseudo-LiDAR based methods that utilize image data to detect 3D information show great prospects for their high cost-effectiveness. However, existing Pseudo-LiDAR methods tend to be intensive computations for the adaption of complex stereo matching algorithms, and therefore can not meet the real-time requirement. To tackle this issue, we propose a lightweight Pseudo-LiDAR 3D detection system that achieves both high accuracy and high responsiveness. Specifically, we adopt a highly efficient depth estimator with Binary Neural Networks (BNNs) to exploit timely depth prediction. To mitigate the accuracy degradation issue caused by the quantization of the BNNs, we introduce several promoting schemes into the depth estimation procedure, including continuous depth approximation, geometric constraints of virtual planes, and training scope restriction, to enhance its performance. Moreover, we further provide an in-depth analysis to the architecture of 3D detector orienting to the Pseudo-LiDAR task and develop effective modules to gain solid accuracy improvement while maintaining low runtime demand. Experiments on the KITTI benchmark and online leaderboard show that our system can conduct the 3D detection within a mere 36 ms, while simultaneously achieving competitive results when compared to the state-of-the-art algorithms.