Pavement crack detection is of great significance for road maintenance. However, the complexity of road surfaces and the irregularity of cracks make it difficult to accurately detect crack regions. We propose a crack detection method based on structural features for the patch-wise crack detection. The novelty of this method lies on the fusion of the local patches in a multi-staged strategy. Deep supervision learning is further used to learn these features at each stage. The fusion features model the structural relevance among cracks. The experimental results prove the effectiveness of our method on the dataset collected from the industrial environments. Among these state-of-the-art methods we compared, our model achieved the best experimental results with an AP 86.97%.