In this paper, we conceive an energy-efficient power and subcarrier allocation scheme for downlink OFDMA systems under average delay constraints. The services associated with multi-users have different packet arrivals and delay requirements. The problem of dynamic power and subcarrier allocation is formulated as a constrained Markov decision process (CMDP) with control actions based on the joint states of channel state information (CSI) and the queue state information (QSI). An online learning algorithm is developed for solving the CMDP problem with the aid of stochastic approximation and the value function approximation approach.