The cut-in behavior of adjacent vehicles presents a challenge for the Adaptive Cruise Control (ACC) system. Inability to proactively discern adjacent vehicles’ cut-in actions could impact driving safety. In addition, abrupt changes in ego vehicle’s following target might provoke excessive reactions, undermining passenger comfort. To address this challenge, this paper integrates trajectory prediction model into a deep reinforcement learning(DRL)-based car-following policy. Utilizing Finite State Machine(FSM), we proactively identify cut-in vehicles based on the predicted trajectories to enhance safety. In designing the DRL-based car-following policy, we propose a novel reward function by analyzing human driving data distribution and considering lateral information of cut-in vehicles. This method enhances driving comfort by significantly reducing abrupt maneuvers in both car-following and cut-in scenarios. Additionally, we investigate the impact of state observation configurations on the performance of the DRL policy. Our experimental findings reveal that incorporating the ego vehicle’s acceleration into the observation state contributes to optimizing comfort and enhancing robustness in scenarios where the observation of other vehicles’ motion state is not precise.