We consider the problem of minimizing a risk measure of the total cost of a Markov decision process (MDP), under the risk-aware MDPs paradigm. This model accounts for the variation/spread/dispersion of the random cost in contrast to classical MDPs which are risk-neutral and emphasize expected cost. In this paper, we extend previous work on risk-aware MDPs by considering a wider class of risk measures which are amenable to dynamic programming. We develop solution methods for this class using grid search and convex approximation schemes, and show that the proposed methods produce the optimal policy. We conclude with numerical experiments which demonstrate the versatility and effectiveness of our approach.