Hybrid precoding is considered as a promising candidate to balance between the sum rate and the power and cost in the mmWave MIMO systems. However, it still suffers from huge power consumption due to considerable radio frequency links and phase shifters. In this study, an energy-efficient adaptive switching module (ASM) and phase modulation array (PMA) assisted hybrid precoding scheme is proposed, in which ASM is introduced to enhance the energy efficiency while PMA is employed to implement analog precoding to reduce power consumption. The ASM-PMA hybrid precoder design problem related to three matrix variables is formulated. Besides, the joint ASM-PMA hybrid precoder/combiner design at both transmitter and receiver is also investigated. Owing to the tough task of joint optimization of matrix variables, we can divide it into two sub-problems. Correspondingly, alternating optimization hybrid precoding (AOHP) algorithm is suggested to solve the PMA. Then, matching theory and convolutional neural network (CNN) based algorithms are respectively proposed to seek the optimal ASM. For the CNN based scheme, relaxed adaptive switching CNN (RAS-CNN) is designed to solve the binary integer problem in the offline train stage, while in the online deployment stage, ideal adaptive switching CNN (IAS-CNN) is adopted. Furthermore, CNN accepts estimated channel as input to yield robust precoding performance and lower computational complexity. Simulation results verify that, the proposed hybrid precoding can outperform other schemes in energy efficiency with satisfactory spectral efficiency.