Beam pumping units (BPUs) are key equipment in oilfield production. Currently, many fault diagnosis methods for BPUs have been developed, and most of them are based on feature or image classification of indicator diagrams. However, low-quality monitoring data and the limited proportion of effective pixels in indicator diagram greatly restrict the performances of these methods. This article proposes an efficient two-step fault diagnosis method for BPUs. In the first step, to overcome the impact of low-quality monitoring data, a dynamic time warping-based matching method is proposed to extract the period of the data, and then a physical model driven method optimized by Bayesian gradient descent is proposed to reconstruct the data. In the second step, to overcome the impact of the limited proportion of effective pixels in indicator diagram, a parallel deep network is proposed which directly takes the time series of the displacement and the load of BPUs as the inputs. Extensive experiments on dataset from 45 real oil wells have shown that, the proposed method can achieve the best performance compared with the state-of-the-art methods, meanwhile the computational load is only 5% of other deep learning-based methods.