In this paper, a sparse binocular fusion convolution neural network is proposed to evaluate the quality of stereo image. In order to simulate the long-term fusion and processing of the left and right views in the brain visual pathway, the network combines the two views of the stereo image four times, and the information processing is carried out through convolution together with the fusion operation. In addition, in order to overcome the computational-intensive and memory-intensive problems of convolution neural networks, a structural sparsity learning (SSL) method is used to regularize the proposed convolution neural network. The experimental results demonstrate that our proposed method performs effectively and efficiently. And the proposed method can achieve 2.0× speedups on LIVE I database and 2.3× speedup on LIVE II database on the basis of improved performance.