In this paper, we present a novel method to generate a dynamic 3D human actor using a time-of-flight depth camera, which provides synchronized color and depth images in real time. In practice, depth data obtained from the depth camera have critical problems, such as optical noise, absence of depth information for certain regions, and unmatched boundaries. In this work, we significantly enhance the visual quality of raw depth images of the human actor by performing noise reduction, recovery of the lost hair region, and boundary refinement. The 3D surface of the human actor is then created by applying a 3D mesh triangulation technique on the enhanced depth images. Finally, we generate a dynamic 3D human actor by providing the surface with corresponding color images. Experimental results show that the proposed method produces a natural and immersive 3D human actor successfully while minimizing problems inherent in the current depth camera-based images.