In the past decade, a lot of efforts are devoted to the researches of trajectory tracking of underactuated water surface robots (or boats, surface vessels, etc.), and a batch of controllers are available now for this challenging problem. However, most, if not all, of the existing trajectory tracking controllers of the underactuated water surface robots assume the global positions of the robots can be measured. In practice, the global position measurement systems are sometimes unstable or even unavailable in the working environments of the robots. To avoid the direct position measurement, this paper presents a new controller for the trajectory tracking of underactuated water surface robots by using visual feedback. This controller works on the basis of a novel adaptive algorithm for estimating global position of the robot online using natural visual features measured by a vision system, and its orientation and velocity measured by AHRS (Attitude and Heading Reference System, IMU&Compass) and velocity sensor. It is proved by Lyapunov theory that the proposed adaptive visual servo controller gives rise to asymptotic tracking of a desired trajectory and convergence of the position estimation to the actual position. Simulations are conducted to validate the effectiveness and robust performance of the proposed controller.