Precise hand segmentation is crucial for gesture-based Human-Machine Interaction. Skin color based hand segmentation using skin color models shows poor performance in complex background where similar colors of the skin and non-uniform illumination exist. We propose a new method for hand segmentation by using an adaptive skin color model and the background information around the hand. Firstly, our method captures pixel values of the hand and the background then converts them into YCbCr color space. Secondly, skin and background Gaussian models based on the color space of CbCr are proposed. Lastly, these models are taken to segment the whole image respectively, and then required for the intersection. The main contribution of the paper is that the background information is taken into account to split image in reversed side to enhance the performance. Experimental results show that our method outperforms the method that uses the skin color model only.