Loading [a11y]/accessibility-menu.js
A Convex Optimization Approach to Distributionally Robust Markov Decision Processes With Wasserstein Distance | IEEE Journals & Magazine | IEEE Xplore

A Convex Optimization Approach to Distributionally Robust Markov Decision Processes With Wasserstein Distance


Abstract:

We consider the problem of constructing control policies that are robust against distribution errors in the model parameters of Markov decision processes. The Wasserstein...Show More

Abstract:

We consider the problem of constructing control policies that are robust against distribution errors in the model parameters of Markov decision processes. The Wasserstein metric is used to model the ambiguity set of admissible distributions. We prove the existence and optimality of Markov policies and develop convex optimization-based tools to compute and analyze the policies. Our methods, which are based on the Kantorovich convex relaxation and duality principle, have the following advantages. First, the proposed dual formulation of an associated Bellman equation resolves the infinite dimensionality issue that is inherent in its original formulation when the nominal distribution has a finite support. Second, our duality analysis identifies the structure of a worst-case distribution and provides a simple decentralized method for its construction. Third, a sensitivity analysis tool is developed to quantify the effect of ambiguity set parameters on the performance of distributionally robust policies. The effectiveness of our proposed tools is demonstrated through a human-centered air conditioning problem.
Published in: IEEE Control Systems Letters ( Volume: 1, Issue: 1, July 2017)
Page(s): 164 - 169
Date of Publication: 05 June 2017
Electronic ISSN: 2475-1456

Funding Agency:


References

References is not available for this document.