The privacy of social media users is a major concern when the users share their content to the public. Sensitive information such as the location of the users can be inferred from relevant content without arising the awareness of the users. With blooming services provided by social media platforms, the users have more freedom to share information via diverse data formats. The multi-modality of the shared information may, in return, worsen the private information leakage caused by inference attacks. In this article, we first examine the problem of location inference on multi-modal data comprised of textual information and visual content. It is observed that the visual content, such as photos shared by social media users, can significantly boost the success rate of location inference. To thwart adversaries who are driven by visual-related data, we propose a defence that mitigates the threat of location privacy breach under an imperceptible utility loss. Our defence, namely LocGuard, perturbs the photos in a one-off manner before sharing them. The perturbations, along with a simple but effective bipartite perturbation strategy, ensure that LocGuard is resistant to adaptive adversaries who can perform adversarial training based on the perturbed photos. Moreover, LocGuard remains effective against open-set adversaries whose data categories in the training dataset are hidden from the defender. In the evaluation, we conduct extensive experiments based on real-world datasets and compare our work with previous methods. The results show that LocGuard significantly outperforms the existing defences. In particular, LocGuard not only achieves better privacy protection and utility preservation for image sharing, but also can effectively defend against adversarial-training-capable attackers.