Skin color is one of most important source of information widely used in Human-Computer Interaction (HCI) system where detecting the presence of the human plays a key role. It is because of the dynamics and articulations of human body which is hard to be captured rather than using primitive feature like skin color. However, skin color varies across different persons limiting us from building a generative skin model for all persons. Here, we propose an efficient skin color detection and tracking based on a face detector to refine the skin model for a specific person and a tracking method using a simple data association algorithm. Beyond that, we also introduce a robust gesture recognition method using different extracted information such as local features, hand motion, head orientation, and finger tips. The final framework is proposed to interpret our gesture recognition method in interacting with a computer through multiple applications: controlling computer mouse and playing games.