Leakage detection of gas pipelines has long been a challenging issue in industries due to the difficulty of learning representation, especially for small leakage. This paper proposes a model based on convolutional neural network (CNN) to identify small leakage. Firstly, in order to gather acoustic signals of small leakage, a gas pipeline system is built to control gas with adjustable pressure. Small leakage is imitated by man-made hole that does not influence the gas pressure, which is denoted as small leakage. Secondly, collected acoustic signals are transformed into frequency domain as pretreatment. Finally, unlike traditional detection techniques with two steps (feature extraction and classification), we establish a CNN-based model with multiple convolutional and pooling layers. In the model, the convolutional and pooling layers are intended to learn the representations of transformed acoustic signals, while a Softmax layer in the last layer is developed to further identify the representation from the front layers. Experiments show the effectiveness of the proposed model with real-world data. Compared with traditional CNN models, the proposed model is less time-consuming and more effective.