In this paper, a distributed, minimum overhead power control algorithm for underlay cognitive radio networks (CRNs) having multiple primary and secondary users is proposed. The problem is formulated as a noncooperative game and a learning algorithm is proposed for optimizing the power allocation of secondary users. In the considered network, secondary users (SUs) do not have full information on the interference and power control strategies of other SUs. As a result, they update their strategy using a simple feedback from the primary user base station that provides the total interference. Although there is no cooperation among secondary users, it is shown that, under incomplete information, the proposed learning algorithm converges to the strategy of the players in the Nash equilibrium of the complete information case. The Nash equilibrium point is derived analytically, and then it is demonstrated that, although each user individually tries to maximize its own payoff, at the end, the proposed algorithm will converge to the complete information game Nash equilibrium point. It is also shown that the algorithm will be capable of adapting to a time-varying environment if some conditions on the SUs' processing power are satisfied. This is due to the slotted time assumption of the algorithm. Simulation results are then used to corroborate the analytical derivations.