Multi-UAV autonomous formation task is one of the important research hotspots in UAV application. A typical mission scenario is constructed for the autonomous formation, maintenance and obstacle avoidance tasks of multi-UAV. Based on Multiple-Agents Deep Deterministic Policy Gradient (MADDPG) algorithm, we designed a hybrid reward distribution mechanism for autonomous formation task, which effectively solved the problem of uneven global reward distribution and individual "selfish strategy", and designed a dynamic communication strategy inside the UAV formation to reduce the computational complexity. It can effectively improve the communication efficiency between UAVs. After training, the UAV can effectively avoid the no-fly zone and efficiently perform autonomous formation flight task. The introduction of the hybrid reward mechanism improves the stability of the UAV autonomous formation task and has certain application prospects.