Abstract
Surface quality control is a crucial part of rail manufacturing. Deep neural networks have shown impressive accuracy in rail surface defect segmentation under the assumption that the test images have the same distribution as the training images. However, in practice detection, the rail images exhibit variations in appearance and scale for different rail types and production conditions. Directly deploying the deep neural network on unseen images shows a performance degradation due to the distribution discrepancies of training images. To this end, we propose a cross-scale fusion and domain adversarial network (CFDANet) to improve the generalization ability of deep neural networks on unseen datasets. To alleviate the domain shift caused by defect scale differences, we design a dual-encoder to extract multi-scale features from images of different resolutions. Then, those features are adaptively fused through a cross-scale fusion module. For the domain shift caused by inconsistent rail appearance, we introduce transferable-aware domain adversarial learning to extract domain invariant features from different datasets. Moreover, we further propose a transferable curriculum to suppress the negative impact of images with low transferability. Experimental results show that our CFDANet can accurately segment defects in unseen datasets and surpass other state-of-the-art domain generalization methods in all five target domain settings. The source code is released at https://github.com/dotaball/railseg_dg.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets used in this study is described in the text. We have provided the source code at https://github.com/dotaball/railseg_dg.
References
Badmos, O., Kopp, A., Bernthaler, T., & Schneider, G. (2020). Image-based defect detection in lithium-ion battery electrode using convolutional neural networks. Journal of Intelligent Manufacturing, 31(4), 885–897. https://doi.org/10.1007/s10845-019-01484-x
Ben Gharsallah, M., & Ben Braiek, E. (2021). Computer aided manufacturing method for surface silicon steel inspection based on an efficient anisotropic diffusion algorithm. Journal of Intelligent Manufacturing, 32(4), 1025–1041. https://doi.org/10.1007/s10845-020-01601-1
Ben-David, S., Blitzer, J., Crammer, K., & Pereira, F. (2006). Analysis of representations for domain adaptation. Advances in Neural Information Processing Systems, 19, 137–144.
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. Proceedings of the 26th Annual International Conference on Machine Learning. https://doi.org/10.1145/1553374.1553380
Chen, C., Bai, W., Davies, R. H., Bhuva, A. N., Manisty, C. H., Augusto, J. B., Moon, J. C., Aung, N., Lee, A. M., Sanghvi, M. M., Fung, K., Miguel Paiva, J., Petersen, S. E., Lukaschuk, E., Piechnik, S. K., Neubauer, S., & Rueckert, D. (2020). Improving the generalizability of convolutional neural network-based segmentation on CMR images. Frontiers in Cardiovascular Medicine, 7, 105. https://doi.org/10.3389/fcvm.2020.00105
Gan, J., Li, Q., Wang, J., & Yu, H. (2017). A hierarchical extractor-based visual rail surface inspection system. IEEE Sensors Journal, 17(23), 7935–7944. https://doi.org/10.1109/JSEN.2017.2761858
Ganin, Y., & Lempitsky, V. (2015). Unsupervised domain adaptation by backpropagation. International Conference on Machine Learning (pp. 1180–1189). PMLR.
Gao, S., Zhou, H., Gao, Y., & Zhuang, X. (2022). Joint modeling of image and label statistics for enhancing model generalizability of medical image segmentation. arXiv preprint arXiv:2206.04336.
Hao, R., Lu, B., Cheng, Y., Li, X., & Huang, B. (2021). A steel surface defect inspection approach towards smart industrial monitoring. Journal of Intelligent Manufacturing, 32(7), 1833–1843. https://doi.org/10.1007/s10845-020-01670-2
Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations.
Jain, S., Seth, G., Paruthi, A., Soni, U., & Kumar, G. (2020). Synthetic data augmentation for surface defect detection and classification using deep learning. Journal of Intelligent Manufacturing, 33(4), 1007–1020. https://doi.org/10.1007/s10845-020-01710-x
Khandelwal, P., & Yushkevich, P. (2020). Domain generalizer: A few-shot meta learning framework for domain generalization in medical imaging. Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning (pp. 73–84). Springer. https://doi.org/10.1007/978-3-030-60548-3_8
Li, D., Yang, Y., Song, Y. Z., & Hospedales, T. (2018). Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v32i1.11596
Liu, Q., Dou, Q., & Heng, P. A. (2020). Shape-aware meta-learning for generalizing prostate MRI segmentation to unseen domains. International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 475–485). Springer. https://doi.org/10.1007/978-3-030-59713-9_46
Luo, Y., Zheng, L., Guan, T., Yu, J., & Yang, Y. (2019). Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2019.00261
Ni, X., Ma, Z., Liu, J., Shi, B., & Liu, H. (2021). Attention network for rail surface defect detection via consistency of intersection-over-union (IoU)-guided center-point estimation. IEEE Transactions on Industrial Informatics, 18(3), 1694–1705. https://doi.org/10.1109/TII.2021.3085848
Nieniewski, M. (2020). Morphological detection and extraction of rail surface defects. IEEE Transactions on Instrumentation and Measurement, 69(9), 6870–6879. https://doi.org/10.1109/TIM.2020.2975454
Niu, M., Song, K., Huang, L., Wang, Q., Yan, Y., & Meng, Q. (2020). Unsupervised saliency detection of rail surface defects using stereoscopic images. IEEE Transactions on Industrial Informatics, 17(3), 2271–2281. https://doi.org/10.1109/TII.2020.3004397
Niu, M., Wang, Y., Song, K., Wang, Q., Zhao, Y., & Yan, Y. (2021). An adaptive pyramid graph and variation residual-based anomaly detection network for rail surface defects. IEEE Transactions on Instrumentation and Measurement, 70, 1–13. https://doi.org/10.1109/TIM.2021.3125987
Pan, F., Shin, I., Rameau, F., Lee, S., & Kweon, I. S. (2020). Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR42600.2020.00382
Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010). Adapting visual category models to new domains. European conference on computer vision (pp. 213–226). Springer. https://doi.org/10.1007/978-3-642-15561-1_16
Singh, S. A., & Desai, K. A. (2022). Automated surface defect detection framework using machine vision and convolutional neural networks. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-021-01878-w
Song, G., Song, K., & Yan, Y. (2020). Saliency detection for strip steel surface defects using multiple constraints and improved texture features. Optics and Lasers in Engineering, 128, 106000. https://doi.org/10.1016/j.optlaseng.2019.106000
Song, K., Wang, J., Bao, Y., Huang, L., & Yan, Y. (2022). A novel visible-depth-thermal image dataset of salient object detection for robotic visual perception. IEEE/ASME Transactions on Mechatronics. https://doi.org/10.1109/TMECH.2022.3215909
Song, Y., Liu, Z., Wang, J., Tang, R., Duan, G., & Tan, J. (2021). Multiscale adversarial and weighted gradient domain adaptive network for data scarcity surface defect detection. IEEE Transactions on Instrumentation and Measurement, 70, 1–10. https://doi.org/10.1109/TIM.2021.3096284
Tao, A., Sapra, K., & Catanzaro, B. (2020). Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:2005.10821.
Tsai, Y. H., Hung, W. C., Schulter, S., Sohn, K., Yang, M. H., & Chandraker, M. (2018). Learning to adapt structured output space for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2018.00780
Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2579–2605.
Vu, T. H., Jain, H., Bucher, M., Cord, M., & Pérez, P. (2019). Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2019.00262
Wang, J., Li, Q., Gan, J., Yu, H., & Yang, X. (2019). Surface defect detection via entity sparsity pursuit with intrinsic priors. IEEE Transactions on Industrial Informatics, 16(1), 141–150. https://doi.org/10.1109/TII.2019.2917522
Woo, S., Park, J., Lee, J. Y., & Kweon, I. S. (2018). CBAM Convolutional block attention module. European on Computer Vision (pp. 3–19). Springer. https://doi.org/10.1007/978-3-030-01234-2_1
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021). SegFormer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 12077–12090.
Yu, H., Li, Q., Tan, Y., Gan, J., Wang, J., Geng, Y. A., & Jia, L. (2018). A coarse-to-fine model for rail surface defect detection. IEEE Transactions on Instrumentation and Measurement, 68(3), 656–666. https://doi.org/10.1109/TIM.2018.2853958
Zhang, L., Wang, X., Yang, D., Sanford, T., Harmon, S., Turkbey, B., Roth, H., Myronenko, A., Xu, D., Xu Z., (2019). When unseen domain generalization is unnecessary? Rethinking data augmentation. arXiv preprint arXiv:1906.03347.
Zhang, D., Song, K., Wang, Q., He, Y., Wen, X., & Yan, Y. (2020). Two deep learning networks for rail surface defect inspection of limited samples with line-level label. IEEE Transactions on Industrial Informatics, 17(10), 6731–6741. https://doi.org/10.1109/TII.2020.3045196
Zhang, H., Song, Y., Chen, Y., Zhong, H., Liu, L., Wang, Y., Akilan, T., & Jonathan Wu, Q. M. (2021a). MRSDI-CNN: Multi-model rail surface defect inspection system based on convolutional neural networks. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2021.3101053
Zhang, S., Zhang, Q., Gu, J., Su, L., Li, K., & Pecht, M. (2021b). Visual inspection of steel surface defects based on domain adaptation and adaptive convolutional neural network. Mechanical Systems and Signal Processing, 153, 107541. https://doi.org/10.1016/j.ymssp.2020.107541
Zhao, X., Sicilia, A., Minhas, D. S., O’Connor, E. E., Aizenstein, H. J., Klunk, W. E., Tudorascu, D. L., & Hwang, S. J. (2021). Robust white matter hyperintensity segmentation on unseen domain. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). https://doi.org/10.1109/ISBI48211.2021.9434034
Zhou, K., Yang, Y., Qiao, Y., & Xiang, T. (2021). Mixstyle neural networks for domain generalization and adaptation. arXiv preprint arXiv:2107.02053.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant No. 51805078), the Fundamental Research Funds for the Central Universities (Grant No. N2103011), the Central Guidance on Local Science and Technology Development Fund (Grant No. 2022JH6/100100023), and the 111 Project (Grant No. B16009).
Author information
Authors and Affiliations
Contributions
SM: Conceptualization, Methodology, Software, Writing—original draft. KS: Project administration, Resources. MN: Investigation, Data curation. HT: Visualization, Writing—review & editing. YY: Supervision, Funding acquisition.
Corresponding authors
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ma, S., Song, K., Niu, M. et al. Cross-scale fusion and domain adversarial network for generalizable rail surface defect segmentation on unseen datasets. J Intell Manuf 35, 367–386 (2024). https://doi.org/10.1007/s10845-022-02051-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-022-02051-7