In this work, we introduce a learning-based method to achieve high-quality reconstructions for inverse scattering problems (ISPs). Particularly, the proposed method decouples the full-wave reconstruction model into two steps, including coarse imaging of dielectric profiles by the back-propagation scheme, and a resolution enhancement of coarse results as an image-to-image translation task solved by a novel perceptual generative adversarial network (PGAN). A perceptual adversarial (PA) loss, which is defined as a perceptual loss for the generator network using hidden layers from the discriminator network, is employed as a structural regularization in PGAN. The PA loss is further combined with the pixel-wise loss, and also possibly the adversarial loss, to enforce a multi-level match between the reconstructed image and its reference one. The adversarial training of the generator and discriminator networks ensures that the structural features of targets are dynamically learned by the generator. Numerical tests on both synthetic and experimental data verify that the proposed method is highly efficient and it achieves superior imaging results compared to other data-driven methods. The validation of the proposed PGAN on ISPs also provides a fast and high-precision way for solving other physics-related imaging problems.