To date, commercial fully autonomous driving is not realized, while level 3 is the next step in the development of autonomous driving. At level 3, the vehicle is driving under the control of the machine, but when feature requests, human driver must take over control. Therefore, autonomous driving control should consider not only efficiency and safety but also human driver’s acceptance. This paper develops a car-following (CF) model as a longitudinal control strategy for level 3 autonomous driving based on the automating entropy adjustment on Tsallis actor-critic (ATAC) algorithm. 1641 pairs of CF trajectories extracted from the Next Generation Simulation (NGSIM) data are applied to train the reinforcement learning (RL) agent. Based on the empirical data distributions, we use time margin, time gap, and jerk to construct the reward function and testify the proposed CF model’s merits. Simulation results show that the proposed model can enable vehicles to drive safely, efficiently, and comfortably. The proposed model has good stability, and the generated driving behaviors are more acceptable for drivers. This work sheds light on developing a better autonomous driving system from the perspective of human factors.