A control-theoretical solution is presented for the direct adaptive visual control of planar manipulators using a fixed camera, when both camera calibration and robot dynamics are uncertain. The proposed scheme is developed for image-based look-and-move visual systems to allow tracking of a moving target. In order to solve the multivariable parameter adaptive problem, the proposed immersion and invariance (I&I) method is used. The scheme is then combined with the adaptive controller for the manipulator, taking into account its nonlinear dynamics and leading to an overall stable adaptive visual system. Simulations and experimental results are also presented for the proposed strategy.