Cognitive Radio (CR) technology enables secondary users (SUs) to opportunistically access unused licensed spectrum owned by primary users (PUs). It has the potential to significantly enhance communication capacity, which is very critical to the next-generation wireless network design and has attracted intensive attention. One of the key issues in CR communications is to detect spectrum availability. Traditional approaches rely on spectrum sensing techniques to address this problem, which, however, consume considerable energy and time, and require complex prior information from PUs. In this paper, we develop a hierarchical spectrum learning system that takes advantage of the fine-tuned convolutional neural network (CNN) and the gated recurrent unit network (GRU), which is called the dual CNN and GRU (DCG), for spectrum availability prediction. Particularly, this model performs accurate predictions on local spectrum availability for each SU without any prior information of PUs. On the other hand, knowing their spectrum availability does not necessarily enable two SUs to successfully communicate on the same channel. This is a challenging problem and has been largely ignored by previous studies designing learning models for CR communications. Towards this goad, we design an enhanced DCG model called EDCG to enable two SUs to find the same channel to communicate with each other by performing channel selection prediction. The performance of the designed DCG and EDCG models is demonstrated through extensive and thorough simulations. The results show that our designed models achieve high prediction accuracy with limited training overhead.