In this paper, we investigate the hierarchical clustering for dense small cells and devise non-cooperative game-theoretic scheme, with aim at increasing the network throughput and minimizing both cross- and co-tier interference. By studying the distances between the small cells and the requirements of cluster formation, we propose a novel hierarchical clustering scheme for the densely deployed small cells, which consists of two phases, i) an absorption mechanism is designed to form the small cells into clusters; ii) to balance cluster populations, a segmentation algorithm is proposed for the clusters that contains excessive small cells. Within each small cell cluster, the spectrum is split into central frequency and marginal frequency which can be reused in a FFR manner. Then, we formulate the spectrum allocation and interference mitigation issues as a non-cooperative game, in which a game theoretical strategy optimization algorithm based on regret-matching is proposed to reach the correlated equilibrium. Numerical results reveal that our approach can achieve the correlated equilibrium with fast convergence and it is effective in offloading traffic and increasing the system throughput in dense small cell networks.