The actuation and control of mini-scale swimming robots have exhibit promising potential in limited and constrained environments, such as biomedical, micro-assembly and microfluidic applications. However, closed-loop control is still a challenging for the precision and repeatability in those applications. In this paper, a new type of soft-micro swimming robot based on magnetoelastic composite material is designed, and the swimmer can move freely on the water surface controlled by a rotating magnetic field with visual servoing. The visual feedback control strategy is proposed based on Mean Shift Algorithm and Fast Template Matching. The algorithm includes three parts: prediction, template matching, target positioning and template updating. To understand the characteristics of the swimmer, the relationship between swimming speed and magnetic field strength and frequency is verified by the algorithm in experiments. Furthermore, the proposed path following method with visual servoing applied in this control system. The experimental results show that the algorithm runs fast, has high accuracy and good real-time performance, and can provide great potential in mini-scale robot applications.