Configuring uplink-downlink frame structure has played an important role in TD-LTE Heterogeneous Cellular Networks (HCNs), but how to dynamically choose the specific TD-LTE frame structure under the densely deployed scenarios has not been well investigated. In this paper, we proposed a novel dynamic frame configuration scheme which could resolve the above problem with low complexity. Specifically, our proposed scheme includes two steps: firstly, a clustering algorithm is designed to divide the femtocells into different sets according to their locations and available resource; secondly, the frame dynamic selection of each set is modeled as a noncooperative game, whose solution could be obtained by reinforcement learning approach. Simulation results are presented to validate the effectiveness of the proposed scheme.