With the increasing requirements for target classification capabilities in an open scene, various open set recognition algorithms emerge in an endless stream. It becomes urgent to evaluate the recognition performance. However, very few studies have examined this crucial and challenging problem previously. To fill the gap, a new evaluation strategy is proposed in this paper. First, two measurements, Openness and openness based on the number of instances per unknown class are presented to quantify the degree of open scenario. The recognition performance is then evaluated by this unified evaluation criteria across several different perspectives, the number of classes, the number of instances, and the synthesis difficulty, on which the ability of algorithms to classify the known classes and identify the unknown classes can be analyzed quantitatively. Multiple experiments are performed on the real dataset. The experimental results demonstrate the effectiveness of the proposed evaluation strategy.