Distracted driving poses a serious threat to traffic safety and remains a widespread problem, highlighting the crucial need for effective recognition of distracted drivers. However, developing models that can generalize across diverse and changing real-world driving conditions is profoundly challenging. Variations in factors like lighting, weather, vehicle type, and drivers complicate generalization. Addressing this critical limitation is key to building recognition models with robust performance for practical deployment. This study presents a novel two-stage unsupervised domain adaptation framework to tackle the important challenge of recognizing distracted drivers across differing environments. The framework first constructs softly assigned class prototypes capturing underlying data structure by aggregating features locally and reweighting sample-prototype relationships globally, which increases the accuracy of class representations. The framework then aligns the probabilities between test samples and prototypes across source and target domains using soft distributional alignment, reducing domain gaps without explicit labeling of the target data. A growth control function balances prototype alignment with classification and adversarial losses. Experiments on distracted driver and object recognition datasets demonstrate this two-stage approach outperforms previous methods, especially under changing driving environments, which is an important problem distracted driving detection research must overcome to effectively enhance road safety.