This paper concerns the stability analysis of image-based visual servoing control laws with respect to uncertainties on the 3-D parameters needed to compute the interaction matrix for any calibrated central catadioptric camera. In the recent past, research on image-based visual servoing has been concentrated on potential problems of stability and on robustness with respect to camera-calibration errors. Only little attention, if any, has been devoted to the robustness of image-based visual servoing to estimation errors on the 3-D structure. It is generally believed that a rough approximation of the 3-D structure is sufficient to ensure the stability of the control law. In this paper, we prove that this is not always true and that an extreme care must be taken when approximating the depth distribution to ensure stability of the image-based control law. The theoretical results are obtained not only for conventional pinhole cameras but for the entire class of central catadioptric systems as well.