Heterogeneous communication environments and broadcast feature of safety-critical messages bring great challenges to mode selection and resource allocation problem. In this paper, we propose a federated multi-agent deep reinforcement learning (DRL) scheme with action awareness to solve mode selection and resource allocation problem for ensuring quality of service (QoS) in heterogeneous V2X environments. The proposed scheme includes an action-observation-based DRL and a model parameter aggregation algorithm considering local model historical parameters. By observing the actions of adjacent agents and dynamically balancing the historical samples of rewards, the action-observation-based DRL can ensure fast convergence of each agent’ individual model. By randomly sampling historical model parameters and adding them to the foundation model aggregation process, the model parameter aggregation algorithm improves foundation model generalization. The generalized model is only sent to each new agent, so each old agent can retain the personality of its individual model. Simulation results show that the proposed scheme outperforms the comparison algorithms in the key performance indicators.