In this article, the charging control of the energy storage system for the pulse power load accommodation in a shipboard integrated power system (SIPS) is formulated as an optimal control problem. The SIPS is an input-affine nonlinear system with randomness and fast dynamics. The improved twin-delayed deep deterministic policy gradient algorithm -one of the deep reinforcement learning (DRL) algorithms, is proposed to solve this optimal control problem. The proposed DRL-based control solution considers the issues regarding the reward function design and input and ramp rate constraints handling for control variables. The proposed approach linked the optimal control and DRL framework. Test cases demonstrated that we could utilize DRL algorithms to control the nonlinear system with fast dynamics by following the specific reward function design, data sampling, and constraints handling techniques.