This paper presents a real-time vision system for assisting driver during nighttime driving. The proposed system provides the following features: 1) effectively detection and tracking of oncoming and preceding vehicles based on image segmentation and pattern analysis techniques. 2) Robust and adaptive vehicle detection under various illuminated conditions at nighttime urban environments benefited by a novel automatic object segmentation scheme. 3) Providing beneficial information for assisting the driver to perceive surrounding traffic conditions outside the car during nighttime driving. 4) Providing a versatile control strategy for in-vehicle facilities of the autonomous vehicles. 5) Offering real-time traffic event-driven video surveillance machinery for recording evidences of possible traffic accidents. Experimental results demonstrate the feasibility and effectiveness of the proposed system on nighttime driver assistance issues.