This paper presents a novel framework of hybrid camera pose tracking system for outdoor navigation system. Traditional vision based or inertial sensor based solutions are mostly designed for well-structured environment, which is however unavailable for most outdoor uncontrolled applications. Our system combines vision, GPS and 3D inertial gyroscope sensors to obtain accurate and robust camera pose estimation result. The fusion approach is based on our PMM (parameterized model matching) algorithm, in which the road shape model is derived from the digital map referring to GPS absolute road position, and matches with road features extracted from the real image. Inertial data estimates the initial state of searching parameters, and also serves as relative tolerance to stable the pose output. The algorithms proposed in this paper are validated with the experimental results of real road tests under different road conditions.