This paper presents a visual servoing approach for high-speed manipulation with one eye-in-hand high-speed vision sensor. By exploiting the special features of high-speed visual feedback and motion, a direct visual servoing scheme based on a simplified interaction matrix is proposed. The camera's internal parameters are not calibrated. Simplifying the interaction matrix allows decoupling of motions, which improves global convergence, as shown by simulations. By further simplifying the interaction matrix so as to make it depth-independent, estimation of depth information is not needed, making the approach suitable for high-speed setpoint control. Simulations were performed to compare the proposed approach with the conventional approach. As applications of the proposed approach, high-speed ball tracking and peg-and-hole alignment experiments were conducted. The results show that the simplified approach worked well for these kinds of manipulation. This paper mainly focuses on translational control; robot arm pose control is ignored because the rotation component of the interaction matrix is independent of the depth information.