For the image-based visual servoing (IBVS) of a manipulator with an unknown structure, the unavailability of the robot Jacobian matrix impedes the accurate control of the manipulator. To solve this issue, this article proposes a data-driven IBVS (DDIBVS) scheme combining model-free learning, matrix inversion estimation, feature tracking, and joint limits. On the one hand, a data-driven learning algorithm is designed, which enables an estimated end-effector velocity to approach the real one and outputs an estimated robot Jacobian matrix. On the other hand, we consider the desired velocity information of the visual feature to improve the tracking accuracy and design an auxiliary parameter to estimate the inversion operation and address the singularity problem. On this basis, a neural dynamic controller (NDC) is developed, which possesses learning, estimation, and control capabilities. Subsequently, the effectiveness, practicability, and superiority of the proposed method are evaluated through simulations and experiments conducted on a 7-degree-of-freedom (DOF) manipulator for visual servoing tasks.