A control system of multiple unmanned aerial vehicles (multi-UAV) is generally very complex when they complete a task in a closely-cooperative manner, e.g. two UAVs cooperatively transport a package of goods. Multi-agent reinforcement learning (MARL) offers a promising solution for such a complex control. However, MARL heavily relies on trial-and-error explorations, facing a big challenge in gathering real-world training data. Simulation environments are commonly used to overcome this challenge, i.e., a control policy is trained in a simulation environment and then transferred into a real-world system. But there often exists a gap between simulation and reality and thus a successful transfer is not guaranteed easily. The domain randomization method provides a workable way to bridge this gap. Nevertheless, the traditional one used in a policy training often suffers from slow convergence and results in an unstable decision policy. To address these issues, this article proposes an adversarial domain randomization method. It utilizes an adversarial generator as a “nature player” to generate a more reasonable training environment so that the trained decision policy can deal with complex situations. Additionally, we improve the prioritized experience replay method by which we can sample those critical experiences, increasing the convergence speed of a training without decreasing the performance of the trained policy. We apply our method to a real-world task of dual-UAV cooperative transportation, and experiments illustrate its effectiveness compared to traditional ones.