The behavior of systems of stochastically interacting particles, be they molecules comprising a chemical reaction network or multi-robot systems in a stochastic environment, can be described using the chemical master equation (CME). In this paper we extend the applicability of the CME to the case when the underlying system of particles is not well-mixed, by constructing an extended state space. The proposed approach fits into the general framework of approximating stochastic processes by hidden Markov models (HMMs). We consider HMMs where the hidden states are equivalence classes of states of some underlying process. The sets of equivalence classes we consider are refinements of macrostates used in the CME. We construct a series of HMMs that use the CME to describe their hidden states. We demonstrate the approach by building a series of increasingly accurate models for a system of robots that interact in a non-well-mixed manner.