In images and videos of a 3D scene, blur due to camera shake can be a source of depth information. Our objective is to find the shape of the scene from its motion-blurred observations without having to restore the original image. In this paper, we pose depth recovery as a recursive state estimation problem. We show that the relationship between the observation and the scale factor of the motion-blur kernel associated with the depth at a point is nonlinear and propose the use of the unscented Kalman filter for state estimation. The performance of the proposed method is evaluated on many examples.