Accurate mapping of urban subsurface is essential for managing urban underground infrastructure and preventing excavation accidents. Ground-penetrating radar (GPR) is a non-destructive test method that has been used extensively to locate underground utilities. However, existing approaches are not able to retrieve detailed underground utility information (e.g., material and dimensions) from GPR scans. This research aims to automatically detect and characterize buried utilities with location, dimension, and material by processing GPR scans. To achieve this aim, a method for inverting GPR data based on deep learning has been developed to directly reconstruct the permittivity maps of cross-sectional profiles of subsurface structure from the corresponding GPR scans. A large number of synthetic GPR scans with ground-truth permittivity labels were generated to train the inversion network. The experiment results indicated that the proposed method achieved a Mean Absolute Error of 0.53, a Structural Similarity Index Measure of 0.91, and an $R^{2}$ of 0.96.