Skip to main content
SpringerLink
Log in

Metal fracture recognition: a method for multi-perception region of interest feature fusion

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Metal materials play a significant role in modern industrial fields. However, due to their exposure to harsh environments, they are susceptible to fatigue damage, which can lead to the occurrence of metal fractures (MF). Effective and accurate identification for MF types is crucial for preventing fractures during service and developing appropriate repair and maintenance strategies. In this paper, a multi-perception region of interest feature fusion (MRIFF) method based on deep learning is proposed to achieve accurate recognition of MF types. The proposed MRIFF method incorporates three key components – the mechanism exploration (ME) module, the adaptive voting (AVT) strategy, and the average-hybrid-attention (AHA) structure, where the ME module is designed to effectively extract and transfer features in metal fracture scanning electron microscope (MFSEM) images from three base models: VGG16, VGG19 and ResNet50, the AVT strategy is developed to synthesize information about MF extracted from these three base models by automatically adjust the weights of individual base model, and the AHA structure is designed to enhance the recognition ability for MF types by effectively fusing information from different scales. These components work synergistically to result in an efficient and accurate image recognition model for MF types. Experiments on the MFSEM image dataset and several publicly available datasets demonstrate that the proposed MRIFF method not only achieves higher accuracy when facing the recognition tasks of MFSEM image, but also exhibits better generalization performance on other image recognition tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data Availability

Part of data included in this study are available upon request by contact with the corresponding author.

References

  1. Chen J, Mao L (2014) Multi-Fractal Spectrum and their Applications in Metal Fracture Surface Images Feature Extraction. AMM 536–537:241–244. https://doi.org/10.4028/www.scientific.net/AMM.536-537.241

    Article  Google Scholar 

  2. Chen T, Lu S, Fan J (2018) S-CNN: Subcategory-Aware Convolutional Networks for Object Detection. IEEE Trans Pattern Anal Mach Intell 40(10):2522–2528. https://doi.org/10.1109/TPAMI.2017.2756936

    Article  Google Scholar 

  3. Chollet F (2017) Xception: Deep Learning with Depthwise Separable Convolutions. In: 30th Ieee Conference on Computer Vision and Pattern Recognition (Cvpr 2017). Ieee, pp 1800–1807. https://doi.org/10.1109/CVPR.2017.195

  4. Croom BP, Berkson M, Mueller RK et al (2022) Deep learning prediction of stress fields in additively manufactured metals with intricate defect networks. Mech Mater 165(104):191. https://doi.org/10.1016/j.mechmat.2021.104191

    Article  Google Scholar 

  5. Dosovitskiy A, Beyer L, Kolesnikov A et al. (2021) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Preprint at arXiv:2010.11929

  6. Gu Y, Vyas K, Shen M et al (2021) Deep Graph-Based Multimodal Feature Embedding for Endomicroscopy Image Retrieval. IEEE Trans Neural Netw Learn Syst 32(2):481–492. https://doi.org/10.1109/TNNLS.2020.2980129

    Article  Google Scholar 

  7. He K, Zhang X, Ren S et al. (2016) Deep Residual Learning for Image Recognition. In: 2016 Ieee Conference on Computer Vision and Pattern Recognition (Cvpr). Ieee, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  8. Hsu WY, Lin WY (2021) Ratio-and-Scale-Aware YOLO for Pedestrian Detection. IEEE Trans Image Process 30:934–947. https://doi.org/10.1109/TIP.2020.3039574

    Article  Google Scholar 

  9. Hu J, Xu W, Gao B et al (2018) Pattern Deep Region Learning for Crack Detection in Thermography Diagnosis System. Metals 8(8):612. https://doi.org/10.3390/met8080612

    Article  Google Scholar 

  10. Huang G, Liu Z, van der Maaten L et al. (2017) Densely Connected Convolutional Networks. In: 30th Ieee Conference on Computer Vision and Pattern Recognition (Cvpr 2017). Ieee, pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  11. Jaremenko C, Ravikumar N, Affronti E et al (2019) Determination of Forming Limits in Sheet Metal Forming Using Deep Learning. Materials 12(7):1051. https://doi.org/10.3390/ma12071051

    Article  Google Scholar 

  12. Jiang S, Min W, Liu L et al (2020) Multi-Scale Multi-View Deep Feature Aggregation for Food Recognition. IEEE Trans Image Process 29:265–276. https://doi.org/10.1109/TIP.2019.2929447

    Article  MathSciNet  MATH  Google Scholar 

  13. Lecun Y, Bottou L, Bengio Y et al (1998) Gradient-based learning applied to document recognition. PROCEEDINGS OF THE IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  14. Leng L, Li M, Zhang J (2010) Research on denoising method of metal fracture image. Computer Engineering and Applications 46(2):172–174. https://doi.org/10.3778/j.issn.1002-8331.2010.27.048

    Article  Google Scholar 

  15. Li M, Xing D, Wang Y et al. (2018) Metal fracture image classification based on adaptive fusion of multiple features. Pattern Recognition and Artificial Intelligence 31(5):453–461. https://doi.org/10.16451/j.cnki.issn1003-6059.201805007

  16. Li T, Chan YH, Lun DPK (2021) Improved Multiple-Image-Based Reflection Removal Algorithm Using Deep Neural Networks. IEEE Trans Image Process 30:68–79. https://doi.org/10.1109/TIP.2020.3031184

    Article  Google Scholar 

  17. Li Z, Wu W (2019) Recognition method of metal fracture images based on empirical ridgelet-2dpca. Journal of Mechanical Strength 41(4):840–844. https://doi.org/10.16579/j.issn.1001.9669.2019.04.012

  18. Li Z, Liu H, Zhang Z et al. (2021) Learning knowledge graph embedding with heterogeneous relation attention networks. IEEE Transactions on Neural Networks and Learning Systems pp 3961–3973. https://doi.org/10.1109/TNNLS.2021.3055147

  19. Lin M, Chen Q, Yan S (2013) Network in network. Preprint at arXiv:1312.4400

  20. Liu H, Zhang Y (2020) Deep learning based crack damage detection technique for thin plate structures using guided lamb wave signals. Smart Mater Struct 29(1):015–032. https://doi.org/10.1088/1361-665X/ab58d6

    Article  MathSciNet  Google Scholar 

  21. Liu H, Fang S, Zhang Z et al (2021) MFDNet: Collaborative poses perception and matrix Fisher distribution for head pose estimation. IEEE Transactions on Multimedia 24:2449–2460. https://doi.org/10.1109/TMM.2021.3081873

    Article  Google Scholar 

  22. Liu H, Nie H, Zhang Z et al (2021) Anisotropic angle distribution learning for head pose estimation and attention understanding in humancomputer interaction. Neurocomputing 433:310–322. https://doi.org/10.1016/j.neucom.2020.09.068

    Article  Google Scholar 

  23. Liu H, Zheng C, Li D et al (2021) EDMF: Efficient deep matrix factorization with review feature learning for industrial recommender system. IEEE Transactions on Industrial Informatics 18(7):4361–4371. https://doi.org/10.1109/TII.2021.3128240

    Article  Google Scholar 

  24. Liu H, Liu T, Chen Y et al (2022) EHPE: Skeleton cues-based gaussian coordinate encoding for efficient human pose estimation. IEEE Transactions on Multimedia. https://doi.org/10.1109/TMM.2022.3197364

    Article  Google Scholar 

  25. Liu H, Liu T, Zhang Z et al (2022) ARHPE: Asymmetric relationaware representation learning for head pose estimation in industrial human-computer interaction. IEEE Transactions on Industrial Informatics 18(10):7107–7117. https://doi.org/10.1109/TII.2022.3143605

    Article  Google Scholar 

  26. Liu T, Wang J, Yang B et al (2021) NGDNet: Nonuniform Gaussianlabel distribution learning for infrared head pose estimation and on-task behavior understanding in the classroom. Neurocomputing 436:210–220. https://doi.org/10.1016/j.compbiomed.2022.105717

    Article  Google Scholar 

  27. Ma M, Li Z, Chen L et al (2018) Review of research on metal fracture image processing. Failure Analysis and Prevention 13(3):196–202. https://doi.org/10.3969/j.issn.1673-6214.2018.03.011

    Article  Google Scholar 

  28. Mehta S, Rastegari M (2022) MOBILEVIT: LIGHT-WEIGHT, GENERAL-PURPOSE, and MOBILE-FRIENDLY VISION TRANSFORMER. Preprint at arXiv:2110.02178

  29. Moreno R, Gorostegui-Colinas E, Lopez de Uralde P et al. (2019) Towards Automatic Crack Detection by Deep Learning and Active Thermography. In: Rojas I, Joya G, Catala A (Eds.) Advances in Computational Intelligence, Iwann 2019, Pt Ii, vol 11507. Springer International Publishing Ag, pp 151–162. https://doi.org/10.1007/978-3-030-20518-8_13

  30. Naik DL, Kiran R (2019) Identification and characterization of fracture in metals using machine learning based texture recognition algorithms. Engineering Fracture Mechanics 219(106):618. https://doi.org/10.1016/j.engfracmech.2019.106618

    Article  Google Scholar 

  31. Rao Y, Lu J, Lin J et al (2019) Runtime Network Routing for Efficient Image Classification. IEEE Trans Pattern Anal Mach Intell 41(10):2291–2304. https://doi.org/10.1109/TPAMI.2018.2878258

    Article  Google Scholar 

  32. Santur Y, Karakose M, Akin E (2017) A New Rail Inspection Method Based on Deep Learning Using Laser Cameras. In: 2017 International Artificial Intelligence and Data Processing Symposium (Idap). Ieee, pp 1–6. https://doi.org/10.1109/IDAP.2017.8090245

  33. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Preprint at arXiv:1409.1556

  34. Snoek J, Larochelle H, Adams RP (2012) Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems 25. https://doi.org/10.48550/arXiv.1206.2944

  35. Solomon C, Breckon T (2011) Fundamentals of Digital Image Processing: A practical approach with examples in Matlab. John Wiley & Sons

  36. Springenberg JT, Dosovitskiy A, Brox T et al. (2014) Striving for simplicity: The all convolutional net. Preprint at arXiv:1412.6806

  37. Sun Y, Li Z, Yan J (2016) Recognition method of metal fracture images based onWavelet kurtosis and Relevance vector machine. MATECWeb of Conferences 39(02):004. https://doi.org/10.1051/matecconf/20163902004

    Article  Google Scholar 

  38. Szegedy C, Vanhoucke V, Ioffe S et al. (2016) Rethinking the Inception Architecture for Computer Vision. In: 2016 Ieee Conference on Computer Vision and Pattern Recognition (Cvpr). Ieee, pp 2818–2826. https://doi.org/10.1109/CVPR.2016.308

  39. Tan M, Le Q (2021) EfficientNetV2: Smaller Models and Faster Training. In: Meila M, Zhang T (Eds.) International Conference on Machine Learning Vol 139, vol 139. Jmlr-Journal Machine Learning Research, pp 7102–7110. https://doi.org/10.48550/arXiv:2104.00298

  40. Tan M, Le QV (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: 36th International Conference on Machine Learning, ICML 2019, Long Beach, CA, United states, pp 10,691–10,700. https://doi.org/10.48550/arXiv:1905.11946

  41. Tan M, Pang R, Le QV (2020) EfficientDet: Scalable and Efficient Object Detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 10,778–10,787. https://doi.org/10.1109/CVPR42600.2020.01079

  42. Thomas A, Durmaz AR, Straub T et al. (2020) Automated Quantitative Analyses of Fatigue-Induced Surface Damage by Deep Learning. Materials (Basel, Switzerland) 13(15). https://doi.org/10.3390/ma13153298

  43. Wang C, Gao J (2012) Design and construction of a database of major accident cases caused by metal material failure. Metalurgical Analysis 32(1):1–6. https://doi.org/10.13228/j.issn.1000-7571.2012.01.004

  44. Woo S, Park J, Lee JY et al. (2018) CBAM: Convolutional block attention module. In: 15th European Conference on Computer Vision, ECCV 2018, September 8, 2018 - September 14, 2018, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 11211 LNCS. Springer Verlag, pp 3–19. https://doi.org/10.1007/978-3-030-01234-2_1

  45. Xie J, Xu C, Chen G et al (2018) Improving visibility of rear surface cracks during inductive thermography of metal plates using Autoencoder. Infrared Phys Technol 91:233–242. https://doi.org/10.1016/j.infrared.2018.04.016

    Article  Google Scholar 

  46. Yamagiwa K, Izumi S, Sakai S (2004) Detecting method of striation region of fatigue fracture surface using wavelet transform. Journal-Society of Materials Science Japan 53:306–312

    Google Scholar 

  47. Yan J, Zhang B, Zhou M et al (2022) Multi-Branch-CNN: Classification of ion channel interacting peptides using multi-branch convolutional neural network. Comput Biol Med 147(105):717. https://doi.org/10.1016/j.compbiomed.2022.105717

    Article  Google Scholar 

  48. Yan Y, Yang H, Wang C (2004) Non-linear pattern recognition method of metal fracture image. Journal of Northeastern University (Natural Science) 9:884–886

    Google Scholar 

  49. Yang J, Chen B, Wang Y et al (2021) Crack detection in carbide anvil using acoustic signal and deep learning with particle swarm optimisation. Measurement 173(108):668. https://doi.org/10.1016/j.measurement.2020.108668

    Article  Google Scholar 

  50. Zhang Q, Huang N, Yao L et al (2020) RGB-T Salient Object Detection via Fusing Multi-Level CNN Features. IEEE Trans Image Process 29:3321–3335. https://doi.org/10.1109/TIP.2019.2959253

    Article  MATH  Google Scholar 

  51. Zhang YL, Gao LX, Li L (2011) Fracture Images Classification Based on Fractional Cosine Transform and Markov Mode. AMR 311–313:970–973. https://doi.org/10.4028/www.scientific.net/AMR.311-313.970

    Article  Google Scholar 

  52. Zhao X, Zhang H, Zhu G et al (2019) A Multi-Branch 3D Convolutional Neural Network for EEG-Based Motor Imagery Classification. IEEE Trans Neural Syst Rehabil Eng 27(10):2164–2177. https://doi.org/10.1109/TNSRE.2019.2938295

    Article  Google Scholar 

  53. Zhuang F, Qi Z, Duan K et al (2021) A comprehensive survey on transfer learning. Proceedings of the IEEE 109(1):43–76. https://doi.org/10.1109/JPROC.2020.3004555

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 62073056 and 61876029; in part by the Applied Basic Research Program Project of Liaoning Province under Grant 2023JH2/101300207 and in part by the Key Field Innovation Team Project of Dalian under Grant 2021RT14.

Author information

Authors and Affiliations

  1. School of Control Science and Engineering, Dalian University of Technology, Dalian, 116024, China

    Han Yan, Chongquan Zhong, Wei Lu & Yuhu Wu

Authors
  1. Han Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chongquan Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuhu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Lu.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflicts of interest to this work. The people involved in the experiment have been informed and formally accepted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, H., Zhong, C., Lu, W. et al. Metal fracture recognition: a method for multi-perception region of interest feature fusion. Appl Intell 53, 23983–24007 (2023). https://doi.org/10.1007/s10489-023-04795-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04795-y

Keywords

Search

Navigation