Segmentation quality evaluation is an important task in image segmentation. The existing evaluation methods formulate segmentation quality as regression model, and recent Convolutional Neural Network (CNN) based evaluation methods show superior performance. However, designing efficient CNN-based segmentation evaluation model is still under exploited. In this paper, we propose two types of CNN structures such as double-net and multi-scale network for segmentation quality evaluation. We observe that learning the local and global information and considering multi-scale image are useful for segmentation quality evaluation. To train and verify the proposed networks, we construct a novel objective segmentation quality evaluation dataset with large amount of data by combining several proposal generation methods. The experimental results demonstrate that the proposed method obtains larger Linear Correlation Coefficient (LCC) value than several state-of-art segmentation quality evaluation methods.