In this paper, we derive a single-letter expression for the capacity of the finite-state channel (FSC) with delayed output and state feedback by formulating the problem in a stochastic control framework. The resulting capacity expression can be evaluated using dynamic programming. Although the considered channel is a special case of the one investigated by Tatikonda and Mitter, our approach is somewhat different and the resulting capacity expression is significantly simpler. This single-letter characterization of the capacity is the first step in investigating posterior matching-like transmission schemes for the case of FSCs.