Skip to main content
Springer Nature Link
Log in

Image-Based Detection of Structural Defects Using Hierarchical Multi-scale Attention

  • Conference paper
  • First Online:
Pattern Recognition (DAGM GCPR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13485))

Included in the following conference series:

  • 1991 Accesses

Abstract

With improving acquisition technologies, the inspection and monitoring of structures has become a field of application for deep learning. While other research focuses on the design of neural network architectures, this work points out the applicability of transfer learning for detecting cracks and other structural defects. Being a high-performer on the Cityscapes benchmark, hierarchical multi-scale attention [43] also renders suitable for transfer learning in the domain of structural defects. Using the joint scales of 0.25, 0.5, and 1.0, the approach achieves 92% mean intersection-over-union on the test set. The effectiveness of multi-scale attention is demonstrated for class demarcation on large scales and class determination on lower scales. Furthermore, a line-based tolerant intersection-over-union metric is introduced for more robust benchmarking in the field of crack detection. The dataset of 743 images covering crack, spalling, corrosion, efflorescence, vegetation, and control point is unprecedented in terms of quantity and realism.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The dataset is available at https://github.com/ben-z-original/s2ds.

  2. 2.

    Code is available at https://github.com/ben-z-original/detectionhma..

References

  1. Atha, D.J., Jahanshahi, M.R.: Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection. Struct. Health Monit. 17(5), 1110–1128 (2018)

    Article  Google Scholar 

  2. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)

    Article  Google Scholar 

  3. Benz, C., Rodehorst, V.: Model-based crack width estimation using rectangle transform. In: 17th International Conference on Machine Vision and Applications (MVA), pp. 1–5. IEEE (2021)

    Google Scholar 

  4. Bianchi, E., Abbott, A.L., Tokekar, P., Hebdon, M.: Coco-bridge: structural detail data set for bridge inspections. J. Comput. Civil Eng. 35(3), 04021003 (2021)

    Article  Google Scholar 

  5. Borse, S., Wang, Y., Zhang, Y., Porikli, F.: Inverseform: a loss function for structured boundary-aware segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5901–5911 (2021)

    Google Scholar 

  6. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  7. Cha, Y.J., Choi, W., Suh, G., Mahmoudkhani, S., Büyüköztürk, O.: Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput.-Aided Civil Infrastruct. Eng. 33(9), 731–747 (2018)

    Article  Google Scholar 

  8. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  9. Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)

  10. Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)

    Google Scholar 

  11. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  12. Dorafshan, S., Thomas, R.J., Maguire, M.: Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. Constr. Build. Mater. 186, 1031–1045 (2018)

    Article  Google Scholar 

  13. Dorafshan, S., Thomas, R.J., Maguire, M.: Sdnet 2018: an annotated image dataset for non-contact concrete crack detection using deep convolutional neural networks. Data Brief 21, 1664–1668 (2018)

    Article  Google Scholar 

  14. Dosovitskiy, A., et al.: An image is worth 16\(times\)16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  15. Dung, C.V., et al.: Autonomous concrete crack detection using deep fully convolutional neural network. Autom. Constr. 99, 52–58 (2019)

    Article  Google Scholar 

  16. Duy, L.D., Anh, N.T., Son, N.T., Tung, N.V., Duong, N.B., Khan, M.H.R.: Deep learning in semantic segmentation of rust in images. In: Proceedings of the 9th International Conference on Software and Computer Applications, pp. 129–132 (2020)

    Google Scholar 

  17. Forkan, A.R.M., et al.: Corrdetector: a framework for structural corrosion detection from drone images using ensemble deep learning. arXiv preprint arXiv:2102.04686 (2021)

  18. Gao, Y., Mosalam, K.M.: Deep transfer learning for image-based structural damage recognition. Comput.-Aided Civil Infrastruct. Eng. 33(9), 748–768 (2018)

    Article  Google Scholar 

  19. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE (2012)

    Google Scholar 

  20. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT press, Cambridge (2016)

    MATH  Google Scholar 

  21. Guo, Z., Hall, R.W.: Parallel thinning with two-subiteration algorithms. Commun. ACM 32(3), 359–373 (1989)

    Article  MathSciNet  Google Scholar 

  22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  23. Hoskere, V., Narazaki, Y., Hoang, T., Spencer Jr, B.: Vision-based structural inspection using multiscale deep convolutional neural networks. arXiv preprint arXiv:1805.01055 (2018)

  24. Hoskere, V., Narazaki, Y., Hoang, T.A., Spencer, B., Jr.: Madnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. J. Civil Struct Health Monit 10, 757–773 (2020)

    Article  Google Scholar 

  25. Katsamenis, I., Protopapadakis, E., Doulamis, A., Doulamis, N., Voulodimos, A.: Pixel-Level corrosion detection on metal constructions by fusion of deep learning semantic and contour segmentation. In: Bebis, G., et al. (eds.) ISVC 2020. LNCS, vol. 12509, pp. 160–169. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64556-4_13

    Chapter  Google Scholar 

  26. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  27. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  28. Liu, H., Miao, X., Mertz, C., Xu, C., Kong, H.: Crackformer: transformer network for fine-grained crack detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3783–3792 (2021)

    Google Scholar 

  29. Liu, Y., Yao, J., Lu, X., Xie, R., Li, L.: Deepcrack: a deep hierarchical feature learning architecture for crack segmentation. Neurocomputing 338, 139–153 (2019)

    Article  Google Scholar 

  30. Liu, Z., Cao, Y., Wang, Y., Wang, W.: Computer vision-based concrete crack detection using u-net fully convolutional networks. Autom. Constr. 104, 129–139 (2019)

    Article  Google Scholar 

  31. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and pattern recognition, pp. 3431–3440 (2015)

    Google Scholar 

  32. Mohan, A., Poobal, S.: Crack detection using image processing: a critical review and analysis. Alexandria Eng. J. 57(2), 787–798 (2018)

    Article  Google Scholar 

  33. Narazaki, Y., Hoskere, V., Yoshida, K., Spencer, B.F., Fujino, Y.: Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts. Mech. Syst. Signal Process. 160, 107850 (2021)

    Article  Google Scholar 

  34. Ortiz, A., Bonnin-Pascual, F., Garcia-Fidalgo, E., et al.: Vision-based corrosion detection assisted by a micro-aerial vehicle in a vessel inspection application. Sensors 16(12), 2118 (2016)

    Article  Google Scholar 

  35. Ortiz, A., Bonnin-Pascual, F., Garcia-Fidalgo, E., Company, J.P.: Visual inspection of vessels by means of a micro-aerial vehicle: an artificial neural network approach for corrosion detection. In: Robot 2015: Second Iberian Robotics Conference. AISC, vol. 418, pp. 223–234. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27149-1_18

    Chapter  Google Scholar 

  36. Pan, X., Yang, T.: Postdisaster image-based damage detection and repair cost estimation of reinforced concrete buildings using dual convolutional neural networks. Comput.-Aided Civil Infrastruct. Eng. 35(5), 495–510 (2020)

    Article  MathSciNet  Google Scholar 

  37. Pauly, L., Hogg, D., Fuentes, R., Peel, H.: Deeper networks for pavement crack detection. In: Proceedings of the 34th ISARC, pp. 479–485. IAARC (2017)

    Google Scholar 

  38. Perez, H., Tah, J.H., Mosavi, A.: Deep learning for detecting building defects using convolutional neural networks. Sensors 19(16), 3556 (2019)

    Article  Google Scholar 

  39. Petricca, L., Moss, T., Figueroa, G., Broen, S.: Corrosion detection using AI: a comparison of standard computer vision techniques and deep learning model. In: Proceedings of the Sixth International Conference on Computer Science, Engineering and Information Technology, vol. 91, p. 99 (2016)

    Google Scholar 

  40. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  41. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  42. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and pattern recognition, pp. 1–9 (2015)

    Google Scholar 

  43. Tao, A., Sapra, K., Catanzaro, B.: Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:2005.10821 (2020)

  44. Xu, Y., Xiao, T., Zhang, J., Yang, K., Zhang, Z.: Scale-invariant convolutional neural networks. arXiv preprint arXiv:1411.6369 (2014)

  45. Yang, F., Zhang, L., Yu, S., Prokhorov, D., Mei, X., Ling, H.: Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Trans. Intell. Transp. Syst. 21(4), 1525–1535 (2019)

    Article  Google Scholar 

  46. Yang, X., Li, H., Yu, Y., Luo, X., Huang, T., Yang, X.: Automatic pixel-level crack detection and measurement using fully convolutional network. Comput.-Aided Civil Infrastruct. Eng. 33(12), 1090–1109 (2018)

    Article  Google Scholar 

  47. Yuan, Y., Chen, X., Wang, J.: Object-contextual representations for semantic segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 173–190. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_11

    Chapter  Google Scholar 

  48. Zhang, L., Yang, F., Zhang, Y.D., Zhu, Y.J.: Road crack detection using deep convolutional neural network. In: IEEE International Conference on Image Processing (ICIP), pp. 3708–3712. IEEE (2016)

    Google Scholar 

  49. Zhang, T.Y., Suen, C.Y.: A fast parallel algorithm for thinning digital patterns. Commun. ACM 27(3), 236–239 (1984)

    Article  Google Scholar 

  50. Zhao, S., Wang, Y., Yang, Z., Cai, D.: Region mutual information loss for semantic segmentation. arXiv preprint arXiv:1910.12037 (2019)

  51. Zheng, S., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6881–6890 (2021)

    Google Scholar 

  52. Zhou, B., et al.: Semantic understanding of scenes through the ade20k dataset. Int. J. Comput. Vision 127(3), 302–321 (2019)

    Article  Google Scholar 

  53. Zou, Q., Cao, Y., Li, Q., Mao, Q., Wang, S.: Cracktree: automatic crack detection from pavement images. Pattern Recogn. Lett. 33(3), 227–238 (2012)

    Article  Google Scholar 

  54. Zou, Q., Zhang, Z., Li, Q., Qi, X., Wang, Q., Wang, S.: Deepcrack: learning hierarchical convolutional features for crack detection. IEEE Trans. Image Process. 28(3), 1498–1512 (2018)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgment

The authors would like to thank DB Netz AG and Leonhardt, Andrä und Partner (LAP) for providing numerous images as well as their consent to publication. Without them this work would have been impossible.

Author information

Authors and Affiliations

  1. Bauhaus-Universität, Weimar, Germany

    Christian Benz & Volker Rodehorst

Authors
  1. Christian Benz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Volker Rodehorst
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Benz .

Editor information

Editors and Affiliations

  1. TU Dresden, Dresden, Germany

    Björn Andres

  2. University of Bonn, Bonn, Germany

    Florian Bernard

  3. Technical University of Munich, Munich, Germany

    Daniel Cremers

  4. University of Hamburg, Hamburg, Germany

    Simone Frintrop

  5. University of Konstanz, Konstanz, Germany

    Bastian Goldlücke

  6. University of Siegen, Siegen, Germany

    Ivo Ihrke

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Benz, C., Rodehorst, V. (2022). Image-Based Detection of Structural Defects Using Hierarchical Multi-scale Attention. In: Andres, B., Bernard, F., Cremers, D., Frintrop, S., Goldlücke, B., Ihrke, I. (eds) Pattern Recognition. DAGM GCPR 2022. Lecture Notes in Computer Science, vol 13485. Springer, Cham. https://doi.org/10.1007/978-3-031-16788-1_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16788-1_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16787-4

  • Online ISBN: 978-3-031-16788-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics