Despite recent advances in Reinforcement Learning (RL), its applications in real-world engineering systems are still rare. The primary reason is that RL algorithms involve exploratory actions that can lead to system constraint violations. These violations can damage physical systems and even cause safety issues, e.g., battery overheat, robot breakdown, and car crashes, hindering RL deployment in many engineering applications. In this paper, we develop a novel safe RL framework that guarantees safety during learning by exploiting a constraint-admissible set for supervision. System knowledge and recursive feasibility techniques are exploited to construct a state-dependent constraint-admissible set. We develop a new learning scheme where the constraint-admissible set regulates the exploratory actions from the RL agent and simultaneously guides the agent to learn the system constraints with a penalty for control regulation. The proposed safe RL algorithm is demonstrated in an adaptive cruise control example where a nonlinear fuel economy cost function is optimized without violating system constraints. We demonstrate that the safe RL agent is able to learn the system constraints to gradually fade out the control supervisor.