Although the fusion of images and LiDAR point clouds is crucial to many applications in computer vision, the relative poses of cameras and LiDAR scanners often need to be discovered. The general 2D-3D registration pipeline establishes correspondences and performs pose estimation based on the generated matches. However, 2D-3D correspondences are inherently challenging to establish due to the large gap between images and LiDAR point clouds. To this end, we build a bridge to alleviate the 2D-3D gap and propose a practical framework to align LiDAR point clouds to the virtual points generated by images. In this way, the modality gap is converted to the domain gap of point clouds. Furthermore, we propose a registration method with cross-domain feature extraction, frustum classification, and domain-agnostic correspondence pruning to narrow the domain gap to establish 3D-3D correspondences between real and virtual point clouds. Experimental results demonstrate that our image-to-point cloud registration method achieves state-of-theart performance on the KITTI Odometry and Oxford RobotCar datasets.