Next-generation wireless networks face a variety of challenges, including fairness problems and high access efficiency demand. The Media Access Control (MAC) layer plays a key role in improving access efficiency and ensure fairness. In this letter, we propose a new MAC protocol that utilizes multi-agent reinforcement learning (MARL) algorithm based on the multi-agent proximal policy optimization (MAPPO) to address these challenges. However, implementing a centralized training with decentralized execution (CTDE) paradigm in a MAC protocol can lead to signaling overhead issues. Therefore, we designed a joint action estimation method and periodic updating parameters scheme that effectively alleviates the communication overhead associated with CTDE. For comparison, we adopt a fully decentralized framework with low signaling overhead based on independent PPO (IPPO) algorithm. The simulation results indicate that our proposed MAPPO-MAC can outperform CSMA/CA and IPPO-MAC in both throughput and fairness with reduced communication overhead.