In this article, an evolution-guided adaptive dynamic programming (EGADP) algorithm is developed to address the optimal regulation problems for the nonlinear systems. In the traditional adaptive dynamic programming algorithms, policy improvement is typically reliant on the gradient information, according to the first order necessity condition. However, these methods encounter limitations when calculating the gradient information becomes infeasible or system dynamics is not differentiable. In response to this challenge, the evolutionary computation is harnessed by EGADP to search for a superior policy during policy improvement. Therefore, compared with the traditional methods, scenarios that gradient information is unavailable can effectively be handled by EGADP. Additionally, the convergence of the algorithm is proven to enhance the rigorousness of the developed method. Finally, the three simulation experiments with realistic physical backgrounds are conducted to comprehensively demonstrate the effectiveness of the established method from different perspectives.