In this paper, the mixed traffic scenario with human-driven vehicles (HDVs) and connected and autonomous vehicles (CAVs) on freeway is considered. In this partly known nonlinear system, an optimal control algorithm using adaptive dynamic programming (ADP) is proposed to deal with the communication latence and drivers’ reaction time, which can stabilize the system under the influence of dead zone and saturation with minimal cost. There are three contributions in this paper. Firstly, in the used ADP algorithm, a critic neural network (NN) is designed to estimate the optimal value of the cost function, which is updated using online data instead of pre-gathered data. This means that the proposed controller can adapt to different parameters of different systems. Secondly, the reaction time of human driver and the time latence of the V2V communication are considered as the state and input delay of the nonlinear system, by adding the terms of delayed states to the optimal value function, the influence of the time delay can be minimized in the process of the critic NN updating. Thirdly, the saturation and dead zone of actuator are considered, by designing a new utility function of control value, the control value is limited from being out of the expected range. Under this condition, the stabilization of the overall system and the effectiveness of the proposed algorithm is proved and validated by means of simulation results.