The defect detection based on machine vision is of great significance for improving the production efficiency and product quality of photovoltaic cells. The common problem in the field of defect detection is the unbalanced distribution between defective data and defect-free data because of the rarity of defective data. However, Convolutional Neural Network (CNN) cannot achieve better processing for unbalanced data sets. In this paper, we propose a novel feature extraction framework that learns the backbone CNN network parameters, optimizes feature space by metric learning in end-to-end training process, which solves the problem of over-fitting of CNN model on imbalanced and small-scale data sets. To verify the effectiveness of this method, extensive experiments on EL image dataset. The comprehensive results demonstrate that the proposed methods can identify the different type of defects with more accuracy rates than the state-of-the-art methods in several cases.