This paper presents a novel and effective method for localizing the position of a mobile robot in real-time in unknown environments by fusing an omnidirectional vision system and odometry/orientation sensor. Based on a new derivation where the omnidirectional projection can be linearly parameterized by the position of the robot and the unknown structures of natural features in the environment, we propose a simple adaptive algorithm, similar to the Slotine-Li algorithm in model-based adaptive control, to estimate the position of a robot on-line by using the continuously tracked features in image sequence, the robot's velocity and orientation angles measured by encoders and orientation sensor. Using the omnidirectional vision system improves robustness of the algorithm because the problem of limited Field of View (FOV) can be avoided. It is proven that the proposed algorithm leads to convergence of the estimation errors to zero. Simulations and experiments demonstrate superior performance of the proposed algorithm.