This paper considers discrete-time constrained Markov control processes (MCPs) under the long-run expected average cost optimality criterion. For Borel state and action spaces a two-step method is presented to numerically approximate the optimal value of this constrained MCPs. The proposed method employs the infinite-dimensional linear programming (LP) representation of the constrained MCPs. In particular, we establish a bridge from the infinite-dimensional LP characterization to a finite LP consisting of a first asymptotic step and a second step that provides explicit bounds on the approximation error. Finally, the applicability and performance of the theoretical results are demonstrated on an LQG example.