We address the problem of planning in the presence of uncertainty and constraints for teams of unmanned vehicles. The problem is formulated as a Constrained Markov Decision Process (C-MDP). We allow for plans with a non-zero but bounded probability of violating constraints, a quantity that we define as risk and provide a solution technique that keeps the risk below a specified threshold while optimizing reward. We also use the decoupling between the dynamics of individual agents to assume transition independence and use this assumption to reduce the complexity of the problem. We provide representative simulation results to show that our technique achieves high reward while keeping risk bounded.