Reinforcement learning (RL) control for nonlinear dynamical systems has seen increasing interests in recent years. However, these methods have limited practical use due to the lack of safety and stability guarantees of the control policy. In this paper, we employ a control-theoretic approach to the stability of RL-based control. We propose a two-step framework to train a Deep Deterministic Policy Gradient (DDPG) agent regulated on the violations of the Lyapunov conditions. In the first step, our framework leverages neural networks to jointly learn stable system dynamics and an associated control-Lyapunov function (cLf) based on the current control policy. In the second step, a DDPG controller is trained to learn an appropriate control policy that maximizes the reward function and minimizes the violation of the Lyapunov condition based on the current iteration of the cLf. We employ a model of dynamics noise when learning the cLf to improve the exploration of alternative state trajectories. The proposed framework is tested on nonlinear vehicle dynamics in a lane-following highway environment. The experimental results demonstrate the ability of the proposed framework to regulate the control policy on learning stable system trajectories with desired driving characteristics.