We explore the relation between the risk-sensitive exponential (exponential of total cost) and Distributionally Robust Reinforcement Learning objectives, and in doing so, we unify some of the popular Reinforcement Learning algorithms. Such equivalence (I) allows to understand a number of well-known Reinforcement Learning algorithms from a risk minimization perspective and (II) establishes the robustness properties of risk-sensitive exponential objective in the Reinforcement Learning context, which in turn provides a theoretical justification for the robust performance of risk-sensitive Reinforcement Learning algorithms in the literature. The robustness of exponential criteria motivates risk-sensitizing current risk-neutral Reinforcement Learning algorithms using such criteria.