In this paper, we propose an efficient framework for reducing noise and holes in depth map captured with an RGB-D camera. This is performed by applying plane fitting to the groups of points assimilable to planar structures and filtering the curved surface points. We present a new method for finding global planar structures in a 3D scene by combining superpixel segmentation and graph component labeling. The superpixel segmentation is based on not only color information but also depth and normal maps. The labeling process is carried out by considering each normal in given superpixel's clusters. We evaluate the reliability of each plane structure and apply the plane fitting only to true planar surfaces. As a result, our system can reduce the noise of the depth map especially on planar area while preserving curved surfaces. The process is done in real-time thanks to GPGPU acceleration via CUDA architecture.