With increasing traffic congestion and accidents, the Internet of Vehicles (IoV) has become a focus of fifth generation (5G) heterogeneous networks (HetNets) and intelligent transportation systems (ITS). IoV has been studied extensively to improve vehicles, road safety, and efficiency. However, the limited spectrum efficiency and interference caused in the 5G HetNets may require robust technology and architecture. The enabling requirements of IoV in 5G HetNets are elaborated and conceptualized in the article. The deep learning algorithms proposed and used in IoV in 5G HetNets can be achieved to reduce delay and improve transmission reliability. The research progress, advantages, and challenges of deep learning in 5G HetNets IoV are illustrated and explored in detail. Moreover, the overall framework of IoV in 5G HetNets is outlined and compared with the related research, which can become the Foundation for future development of intelligent IoV.