The problem of direct adaptive visual servoing of robot manipulators is considered. A solution is developed for image-based look-and-move visual systems to allow tracking of a desired trajectory, when both camera calibration and robot dynamics are uncertain. In order to solve the multivariable parameter adaptive problem, the recently proposed immersion and invariance (I&I) method is used. The scheme is then combined with a robust motion controller for the manipulator, which takes into account its uncertain nonlinear dynamics and leads to an overall stable adaptive visual servoing system. The effectiveness of the proposed strategy is illustrated through simulations and experimental results.