Time delay, especially varying time delay, is always an important factor affecting the stability to the human-in-the-loop system. Previous research usually focuses on the performance of the internal signal transmission part, but rarely considers the whole system with human and environmental factors comprehensively. For the problem, we investigate this issue by using an improved proportional-derivative-like plus damping (PD-like + d) control method, the derivative term of which is calculated based on the estimation of time delays. An embedded adaptive fuzzy logic systems (FLS)-based observer is developed to estimate and compensate for the errors caused by time delay estimations and uncertain force/ torque measuring errors. The advantages of the proposed control scheme are discussed by building an environmental input energy function and the effectiveness is also verified by the comparative simulations. The results show that under the same simulation conditions, the follower can track the leader's movements well, and the energy introduced into the environment is the same as that of the leader, which means the extra energy is dissipated to enable the object to be manipulated as desired by the leader side.