In Internet of Vehicles Networks, federated learning balances the privacy of driving data with autonomous driving. However, poisoning attacks seriously compromise the security of autonomous driving by undermining training data. In addition, training federated learning concerns require privacy protection of model parameters. To address these problems, we propose the differentially private federated clustering framework. We adopt an innovative approach using federated clustering to mitigate the impact of malicious data introduced by adversarial third parties while ensuring data integrity and security. This approach aims to project driving data into the feature domain and cluster its semantic features. Our framework leverages the consistency of data labels and semantic representation features as the foundation for data sanitization, effectively removing malicious data. We assess the effects and efficiency of our framework. Furthermore, we delve into the prospects and challenges of federated learning in Intelligent Connected Vehicles.