We consider a dense small cell network underlaying the conventional cellular infrastructure, where the macro base station (MBS) activates some small base stations (SBSs) in case it is necessary to offload the macro cell traffic. In order to become activated, every SBS is allocated the required power supply by the MBS. We develop a realistic system model where the MBS does not have any prior information on small cells' characteristics such as local channel qualities or user traffic, and the network's statistical characteristics are time-variant. We model the energy-efficient SBS activation problem as a budget-limited multi-armed bandit game where the reward generating processes of arms are piece-wise stationary. We develop a decision making policy to solve the energy-efficient activation problem, and prove an upper-bound for the regret. The performance of the proposed scheme is evaluated also through numerical analysis.