Clustering ensemble is a momentous technique in machine learning and contribute much to the applications in many areas. General clustering ensemble methods pay more attention to predicting cluster labels than structures of clusters. In fact, learning cluster structures implicates sufficient information to rebuild the dataset and is competent for being the replacement of redundant predicted cluster labels. In this paper, we introduce the fuzzy theory into the structure framework and propose a newfangled double fuzzy c-means structure ensemble framework, named as FCM²SE. FCM²SE makes use of the cluster structure information instead of predicted labels to gain a representative ensemble structure. We also design two novel labeling criteria to distribute the samples to the corresponding clusters. The empirical results on synthetic datasets and UCI machine learning datasets demonstrate the effectiveness of the proposed method.