Foreground segmentation has a critical role in image processing and computer vision-based applications. Extracting accurate foreground pixels, especially in the boundary region is still an unsaturated area in image analysis. The proposed method explains a segmentation scheme to extract human area from an image or from video frames taken by RGB-Depth sensors like Microsoft Kinect camera. The pre-segmentation is achieved by using the depth information. Here in this algorithm, more focus is given for the enhancement of the pre-segmented image for accurate boundary matching of the segmented human factor. Since the depth analysis have limitations in detecting accurate border pixels, post processing steps are involved here for restoring the missing foreground portions and to avoid background pixels from the final segmented output. Initially hair area is restored using Chan-Vese active contour detection along with Viola-Jones face detection. For making the segmentation quality enough to use in multimedia applications like film editing, fine outer boundary estimation is performed using digital matting procedures.