This paper addresses the problem of the low-energy coordinated formation control of multiple unmanned aerial vehicles (UAVs) in a time-varying bilateral teleoperation system. To achieve the multi-UAV cooperative control with time-varying delays, a passive proportional velocity/position errors plus damping injection controller is proposed to enforce the coordinated formation and force tracking of the master haptic device and the slave UAVs. The stability of the controller is analysed by the Lyapunov-Krasovskii function and the delay-dependent stability criteria of system are obtained. Moreover, a min-weighted rigid graph is adopted to reduce the communication energy dissipation. With the optimal rigid topology, the communication links and costs can be decreased. Finally, the human-in-the-loop simulations are performed to evaluate the effectiveness of the proposed control scheme.