We have adopted computer vision and deep learning techniques to obtain detailed information of heavy trucks on the bridge, assisting the bridge structure health monitoring(BSHM) for structural analysis and state assessment. The R-ZF (ZF with residual structure frameworks) algorithm was proposed, where the residual structure was added into the ZF network, and the anchor was optimized. The experimental results showed that the R-ZF method had a mean average precision of 91.51% for Truck and Wheel, and the model recognition speed reached 0.050s/frame. At the same time, this method could obtain the detailed information of heavy trucks on the bridge accurately.