In this paper, the reinforcement learning (RL) approach is proposed to mitigate the derivative voltage spike for the proportional-integral-derivative (PID), fractional-order PID (FOPID), and recently proposed time-varying FOPID (TV-FOPID) controller. The voltage spike is often generated at the output of the PID type controller due to the derivative term, which responds to the instantaneous change of the error caused by the change of the set point. The voltage spike may cause damage to the electric motors, electronic devices, or other components. Therefore, it is important to mitigate the voltage spike at the output of the controller while keeping the step response with acceptable performance. In this paper, two reinforcement learning algorithms, deep deterministic policy gradient (DDPG) and twin-delayed deep deterministic (TD3), are adopted to tune the controller parameters. The simulation results demonstrate that the TV-FOPID controller with RL significantly reduces the voltage spike at the output of the controller while maintaining an acceptable step response.