5G millimeter wave (mmWave) communication is an efficient technique for low delay and high data rate transmission in vehicular networks. Due to the high path loss in 5G mmWave band, 5G base stations need to be densely deployed, which may result in great deployment expenditures. In this letter, we jointly consider a relay selection problem in multihop 5G mmWave device to device (D2D) transmissions and a power level allocation problem of mmWave D2D links. We propose a centralized hierarchical deep reinforcement learning based method to find an optimal solution for the problem. The proposed method does not rely on the information of links, and it tries to find an optimal solution based on the information of vehicles. Simulation results show that the convergence of the proposed method, and the transmission delay performance of proposed method is better than a link-quality-prediction based method, and close to a link-quality-known method.