Regulation service (RS) reserves, a critical type of bi-directional capacity reserves, are provided today by expensive and environmentally unfriendly centralized fossil fuel generators. This paper investigates provision of RS reserves by the demand side. We consider a smart building operator that is capable of modulating the aggregate consumption of the building loads via price signals in response to an unanticipated RS signal that an independent system operator broadcasts. We first model the RS signal and load behavior, and formulate the related stochastic dynamic programming (DP) problem. Then, in order to deal with the complexity of the DP problem resulting from the uncountably infinite allowable price set, we characterize certain key properties of the DP dynamics, solve the DP problem for a discretized price policy to observe the structure of the optimal policy and re-capture the continuous price policy in an analytic approximate policy iteration (API) algorithm using the above properties and structure. We finally provide numerical evidence that the novel API algorithm converges to a continuous dynamic price policy that outperforms optimal discretized price policies in both computational effort and average cost.