Accurate and fast detection of pavement distress can provide reliable and effective technical support for pavement maintenance and rehabitation. Recently, deep learning has been widely used in pavement distress detection. However, its application is still limited by the laborious and difficult annotation process due to the complex topology of pavement distress. In this study, we propose a pavement anomaly detection network (PAD Net), which is a semi-supervised learning approach based on generative adversarial networks for identifying pixel-level anomalous image segments. We build a mapping function for unpaired abnormal and normal pavement images through a framework containing two generators and three novel discriminators. The framework is capable of maintaining background pixels and modifying anomalous foreground regions with the help of multi-style discriminators that consider interrelationships of multi-scale generated images. Meanwhile, pixel-level abnormal areas are detected through an end-to-end mask channel. Experiments show that our approach is able to achieve 80.75% accuracy on our dataset without pixel-level or patch-level annotations. Quantitative comparisons with several prior semi-supervised methods demonstrate the superiority of our approach.