Device-to-Device (D2D) communication and edge caching are two promising approaches to relieve stress on core networks, and the dynamic topology and complex device types of D2D networks bring more challenges to the design of caching strategies. However, most of the existing research works on D2D caching focus on predictable user mobility without considering its randomness. In addition, they ignore the heterogeneity of user devices. In this paper, we focus on the caching strategies in sophisticated dynamic heterogeneous D2D networks with randomly moving users and heterogeneous user devices. We formulate the average user request delay expectation minimization problem which is proved to be NP-hard. To diminish the impact of coupling user strategies and make full use of the network resources, we propose the policy-network assisted multi-agent collaboration (PAMAC) caching strategy, in which the agents are placed on the user devices and trained collaboratively using a distributed approach. Specifically, in PAMAC, we first propose the collaborative training order optimization (CTOO) algorithm to schedule the training order of the user devices in the network. Then we propose the binary particle swarm policy network (BPSP) algorithm and introduce it to each agent to design the multi-agent collaborative caching scheme. Extensive simulations are conducted to show that PAMAC outperforms other existing caching strategies.