With increasingly complex network structure, requirements for heterogeneous 5G are also growing. The aim of this study is to meet the network security performance under the existing high-capacity and highly reliable transmission. In this context, deep learning technology is adopted to solve the security problem of the 5G heterogeneous network. First, the security problems existing in 5G heterogeneous networks are presented, mainly from two aspects of the physical layer security problems and application prospects of deep learning in communication technology. Then the combination of deep learning and 5G heterogeneous networks is analyzed. The combination of deep learning technology, modulation information recognition, and beam formation is introduced. The application of deep learning in communications technology is analyzed, and the modulation information recognition and beamforming based on deep learning are introduced. Finally, the challenges of solving security problems in 5G heterogeneous networks by deep learning are explored. The results show that the deep learning model can solve the modulation recognition problem well, and the modulation mode of the convolutional neural network can well identify the modulation signals involved in the experiment. Therefore, deep learning has a good advantage in solving modulation recognition. In addition, compared to the traditional algorithm, the unsupervised beamforming algorithm based on deep learning proposed in this research can effectively reduce the computational complexity under different numbers of transmitting antennas, which verifies the superiority of the unsupervised beamforming algorithm based on deep learning proposed in this research. Therefore, the present work provides a good idea for solving the security problem of 5G heterogeneous networks.