We formulate the problem of minimizing the operating cost of supplying residential hot water as a discrete-time finite-state Markov decision process. We apply state aggregation to reduce the effective size of the state space and utilize density estimation to obtain an algorithm that is robust to modeling changes in the cost function. We then use approximate policy iteration to obtain an asymptotically optimal solution to the problem when hot water demand is assumed to be independent of previous demand. We also provide heuristics for solving the problem when the demand is not assumed to be independent of its history. To evaluate the performance of the algorithms, we model the thermodynamics of a 20 gal water heater and simulate hot water demand for a typical two-person household. Test results show that when compared to a regular water heater, the ADP solution can reduce water heating costs by as much as 1/3 while maintaining nearly the same level of comfort. In addition, we discuss how the algorithm can be modified to incorporate solar water heating and we consider the related problem of using residential water heaters for load smoothing.