This paper presents a bare-hand controlled augmented reality (AR) map navigation system. A set of symbolic hand gestures is defined as controlling commands. Users can directly move their hands on a real map. When certain hand gestures are recognized the relative geography information, video clip or 3D scene would be played based on some display devices such as the HMD and computer screen. Our system is concentrated on the popular family use. A single Web cam is used as the video input device. A novel hand segmentation method is presented to solve the problems such as light changing and un-uniformed background. Based on this approach a finger finding and hand gestures recognizing algorithm is developed. The performance of the system proved that our methods are robust and effective. And it is easy to configure and use for family users.