Data applications account for the magnitude of traffic generated in the Cellular Networks. To meet the ever-increasing traffic demand, advancement in resource allocation is crucial. Dense Heterogeneous Networks (HetNets) aim at meeting the high data rate requirements of the future 5G communications. The paper's main contribution is to maximize the capacity of 5G dense networks via machine learning type Q-learning control algorithms in the Downlink. For broader comparison, we proposed two Power Allocation Q-learning algorithms, namely Distributed and Formulated. To investigate the impact of femto user equipment (FUE) in the network, the location of macro user equipment (MUE) is considered in the reward function. Furthermore, a cooperative approach is utilized to decrease the time search complexity. This approach is complex and requires further improvement. The obtained simulation results showed that the proposed Distributed algorithm outperformed the Formulated and Cooperative approaches. Accordingly, in the latter approaches the number of served FUEs increased however, at the expenses of the MUE Quality of Service (QoS).