In a noncooperative dynamic game, multiple agents operating in a changing environment aim to optimize their utilities over an infinite time horizon. Time-varying environments allow to model more realistic scenarios (e.g., mobile devices equipped with batteries, wireless communications over a fading channel, etc.). However, solving a dynamic game is a difficult task that requires dealing with multiple coupled optimal control problems. We focus our analysis on a class of problems, named dynamic potential games, whose solution can be found through a single multivariate optimal control problem. Our analysis generalizes previous studies by considering that the set of environment's states and the set of players' actions are constrained, as it is required for many applications. We also show that the theoretical results are the natural extension of the analysis for static potential games. We apply the analysis and provide numerical methods to solve four example problems, with different features each: i) energy demand control in a smart-grid network; ii) network flow optimization in which the relays have bounded link capacity and limited battery life; iii) uplink multiple access communication with users that have to optimize the use of their batteries; and iv) two optimal scheduling games with time-varying channels.