A robust inverse optimal adaptive visual servo controller is described that asymptotically regulates a camera's pose to a desired pose despite uncertainty in the time-varying distance from the camera to the target and parametric uncertainty in the camera calibration matrix. A high gain robust inverse optimal controller asymptotically stabilizes the rotation error system, whereas, an inverse optimal adaptive controller is shown to stabilize the translation error by compensating for the unknown depth and camera calibration parameters. A Lyapunov-based stability analysis is used to examine the stability and an inverse optimal analysis is used to determine the optimality of the developed controller.