Multi-user semantic communication (MUSC) has emerged as a promising paradigm for future 6G networks and applications, where massive clients (e.g., mobile devices) collaboratively construct a global semantic decoder without sharing their local data. However, due to the lack of direct access to clients’ data, MUSC is vulnerable to data poisoning attacks (DPAs), wherein malicious participants send updates derived from poisoned training samples. Current defense techniques against DPAs are designed for traditional networks and are not directly applicable to MUSC. In this paper, we propose an effective attack-defense game framework, denoted as DPAD-MUSC, tailored to defend against DPAs during image transmission for MUSC. First, we determine each attack-type's optimal attack policy based on reinforcement learning, with the aim of strengthening the attack while avoiding detection. To generate adversarial samples accordingly, we devise an adversarial samples generator (ADV-Generator) based on conditional generative adversarial network (CGAN). Then, we introduce an attack defender (DPA-Defender) to detect data poisoning attacks and exclude poisoned samples from the target model's learning process, with the adversarial samples generated under the guidance of the optimal attack policy to enhance the detector's robustness. Simulation results demonstrate that the DPAD-MUSC can find optimal attack policies that cause a greater accuracy drop in the target model while maintaining a higher evasion rate. The ADV-Generator can generate effective adversarial samples and the DPA-Defender outperforms five state-of-the-art methods on three widely used image datasets under additive white Gaussian noise (AWGN) channel in terms of Top-1 accuracy.