UAV-assisted heterogeneous networks (HetNets) in urban environments pose unique challenges in maintaining robust, uninterrupted, and efficient wireless communication. Existing models often struggle to optimize both rate and handover (HO) decisions effectively, particularly in dynamic and densely populated urban scenarios. This paper proposes a novel adaptive Q-learning framework integrated with multi-criteria decision-making techniques to enhance network rate, HO performance, and probability of identification (POI) in urban UAV-assisted HetNets. The proposed methodology addresses the limitations of existing approaches by incorporating dynamic environmental feedback and real-time decision-making, leading to more efficient resource allocation, reduced HO failures, and improved POI. Additionally, the framework leverages a hybrid optimization strategy to enhance energy efficiency (EE) while maintaining adequate quality of service (QoS) for ground users (GUs). The simulation results are thoroughly validated using ray tracing (RT) techniques and benchmarked against current standard models, including 3GPP and NYM, demonstrating the proposed model's superiority in rate, energy efficiency, HO performance, and POI in complex urban environments.