We introduce a spatially-consistent implicit f unction representation (sci-f) for high-resolution volumetric computed tomography (CT) image recovery from uni- and bi-planar X-ray images. We devise a deep end-to-end learning scheme to parameterize the unified implicit function conditioned on the input 2D X-rays and predict the 3D anatomies. Instead of the discretized voxel representation in the existing deep learning-based CT reconstruction from sparse 2D X-rays, the lightweight and memory-efficient sci-f enables a volume recovery with continuous resolutions. The sci-f bridges the 2D pixels with their counterpart voxel through the 3D ray for generalized and spatially consistent 3D reconstruction of anatomical structures, instead of existing per-subject learning of neural representation. Experiments demonstrate the efficacy of the proposed approach in X-ray-based volume recovery from clinically obtained X-rays, with performance gains over state-of-the-art deep X-ray-based CT reconstruction.