The emergence of Intelligent Connected Vehicles (ICVs) heralds a transformation in modes of transportation. This intricate system capitalizes on a wealth of data generated by various terminals, enabling AI models to streamline numerous applications, ranging from travel assistance and urban road monitoring to navigation path planning. Despite this, the data collected from these disparate terminals frequently encapsulates sensitive information pertaining to the entire ICV ecosystem. This can raise concerns about privacy breaches, especially with regard to geographical locations and personal images. Therefore, it is imperative to devise strategies that not only enable intelligent learning and data sharing across the whole scenario but also preserve privacy. In this paper, we introduce SemanticICV, an endogenous secure full-scenario learning framework designed to formulate the semantic logic of ICVs at a foundational level. This semantic abstraction inherently imbues the framework with privacy-preserving attributes. Building upon this, we propose a horizontal semantic sharing method (using federated learning) and a vertical semantic cross-layer linkage method (employing knowledge distillation) integrated within the full-scenario learning. These methods serve to enhance the data sharing and innate privacy protection of the full-scenario. We substantiate the efficacy of our proposed solution through meticulous simulation experiments.