Deep learning has shown remarkable success in data-driven vehicle detection, relying on collected training samples from known scenes. A challenging problem arises when these detectors handle agnostic scenes, while keeping the performance of previous ones. To address this issue, a feasible remedy is to learn a set of domain-adaptive detectors by aligning the features from one scene to another. However, the improvement obtained in this way is inflexible despite the progress in object detection. An important reason is that the memory sizes grow massively with deliberately saving all scenes-independent detectors, while ignoring the relationship among different scenes. In this paper, we aim to bridge the gap between scene diversification and object consistency for scene-aware vehicle detection. Specifically, a novel structured network is proposed to integrate selective assignment of scene-specific parameters into the vehicle detection framework. Extensive experiments conducted on different scenes including BDD, Cityscapes-car, CARPK, etc, demonstrate that the proposed method achieves impressive performance, while keeping the performance of previous scenes as the scene changes.