The cooperation between a manned aerial vehicle (MAV) and an unmanned aerial vehicle swarm (UAVS) to perform tasks will be a new focus of UAVS application in the foreseeable future. It is expected that a UAVS should be able to complete time-varying formation (TVF) flight while simultaneously tracking the trajectory of a MAV. This article studies the tracking-oriented TVF control problem of UAVSs under jointly-connected directed switching topologies (JCDST). To begin with, a distributed formation controller with one intermediate variable for estimating each UAV's state is devised based upon dynamic output feedback control approach. Then, the tracking-oriented TVF formation control issue is converted into an issue about the asymptotic stability control of a fast time averaging system via an appropriate variable conversion. Furthermore, the sufficient condition for the asymptotic stability of the fast time averaging system, and the computing approach of the unknown control matrices are acquired and proved through a special kind of similar matrix transformation and Hurwitz stability criterion. Finally, the effectiveness and correctness of the proposed method are verified by comparing simulation results.