The adverse weather conditions have brought considerable difficulties in vision-based applications, which are closely related to the driving safety of autonomous vehicles. However, to date, the greater part of the existing environmental perception studies are under ordinary conditions, and the method to deal with the adverse weather conditions was ignored. Hence, this paper proposes an all-in-one gated adversarial network (AIO-GAN) to improve the performance of vision-based environment perception algorithms under adverse weather conditions, including in rain, haze, lack of light, etc. Three key technical contributions are made. At first, the deep learning based gated transformer module was proposed to classify the input mixed images to different collections by passing them through different branches. Second, the multi-branch based variational autoencoder-generative adversarial network was proposed to solve the ill-pose problem of the solution. Third, high-level weight sharing encoders was given out to guarantee the stability and the high quality of the training process. In this way, the unified clean-style images can be achieved, even if the mixed multi-modal images are transferred from the source domain of complex weather scenes. Extensive experimental results show that the proposed method has achieved better performance than state-of-the-arts and improved the accuracy of target detection.