In 6G wireless communication scenarios, the ultra-dense network (UDN) scenario has received extensive attention, as it is an inevitable case of 6G networks. How to keep users safe while providing universal service to a number of users is becoming a crucial topic of UDN. To battle this issue, we propose a distributed Q-learning-enabled multi-dimensional spectrum sharing security scheme for the millimeter-wave frequency band. With this scheme, a fine-grained spectrum sharing security framework based on spatial multiplexing is established, which balances UDN performance and security. Aiming at the multi-dimensional spectrum sharing security and optimal allocation problem, non-cooperative game theory is used for modeling and analysis. To prove the validity of this scheme, we used numerical simulations to verify the performance of the proposed distributed Q-learning algorithm for solving the spectrum sharing optimization problem in the UDN scenario. The results show that the proposed multi-dimensional spectrum sharing security scheme can significantly reduce the access delay of users, increase the number of unauthorized users that can be accommodated, and improve throughput, while achieving good security performance of the network.