Next-generation wireless networks require the integration of cognitive networks (CNs) and decision-making techniques to improve the spectrum efficiency. The conventional spectrum sharing schemes require full channel state information of CNs and cannot satisfy the low-latency requirement of next-generation wireless networks. Artificial intelligence has shown its high potential to perform decision-making and improve resource utilization efficiency. Hence, we propose a multi-agent reinforcement learning (MARL)-based scheme for spectrum sharing, which is a promising self-decision technique in highly dynamic and complex wireless networks. We analyze several key challenges when MARL is applied to spectrum sharing, such as multi-objective function formulation, multi-dimensional action space, and partial channel state information. Then, we propose efficient solutions and apply the explainable DRL to improve the convergence efficiency in spectrum sharing. The proposed architecture, fundamentals, and challenges provide a clear vision for MARL in CNs.