Vision-based hand gesture recognition (VGR) systems must provide the following functionalities or criteria to control a computer mouse: (i) hand tracking ability, (ii) continuous static and dynamic hand gesture recognition, and (iii) efficient resource management. Our motivation stems from the fact that only a few research so far has accommodated all these three criteria. In this paper, we developed a VGR system that accommodates these three criteria. We propose an algorithm that simultaneously detects and classifies hand gestures using RGB images and hand skeletons. To evaluate our work, we used the IPN dataset which consists of hand gestures that are suitable for mouse control. Compared to previous methods on the IPN dataset, our resulting VGR system achieves better performance in both isolated and continuous hand gesture recognition (HGR). For continuous HGR, we achieved 61.30% Levenshtein accuracy.