IA is a promising solution for the interference management of future wireless networks. On the other hand, physical layer security is a critical issue of wireless communications in the presence of adversaries. Recently, these two important fields tend to be researched closely together. In this article, some of the key results are summarized, and two primary attacks at the physical layer of IA-based networks, adversarial jamming and eavesdropping, are further studied. We first propose an anti-jamming scheme by aligning the jamming signal together with interference among users cooperatively when an adversarial jammer exists. Then an AN scheme is proposed, in which the external eavesdropping is disrupted by AN without introducing any additional interference to the legitimate network. To further analyze the potential threat, a collusive eavesdropping scheme by some hostile IA users in the network is also proposed. Simulation results are presented to show the effectiveness of these schemes. Finally, some future challenges are also summarized.