Abstract
Deep learning methods, especially convolutional neural networks (CNNs), are widely used for industrial surface defect segmentation due to their excellent performance on visual inspection tasks. However, the problems of overfitting and low generalizability affect the performance of CNN-based surface defect segmentation models. Therefore, data augmentation is necessary to reduce overfitting and improve generalization. However, existing data augmentation methods are random and independent of the downstream defect detection tasks. Thus, we propose a simple plug-and-play data augmentation method based on the requirements of the CNN defect segmentation task. We first pretrain a defect segmentation model on a training set and obtain confidence maps. Then, we occlude high-confidence regions based on the data augmentation module to balance the attention paid by the model to high- and low-confidence regions. Finally, to prevent overfitting, we periodically update the confidence map. We conduct sufficient experiments on the Kolektor surface defect (KSD) metal surface dataset and an optoelectronic chip dataset. The proposed method can make full use of the information contained in the existing datasets to improve the accuracy of the segmentation model (intersection-over-union (IOU) 6.62% improvement over baseline on the KSD dataset and 3.20% on the optoelectronic chip dataset). Moreover, the proposed method is highly applicable to various mainstream defect segmentation methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bissoto, A., Valle, E., & Avila, S. (2021). Gan-based data augmentation and anonymization for skin-lesion analysis: A critical review. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1847–1856).
Chen, J. N., Sun, S., He, J., Torr, P. H., Yuille, A., & Bai, S. (2021). Transmix: Attend to mix for vision transformers. arXiv preprint arXiv:2111.09833.
Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848.
Chen, P., Liu, S., Zhao, H., & Jia, J. (2020). Gridmask data augmentation. arXiv preprint arXiv:2001.04086.
Cheng, K.C.-C., Chen, L.L.-Y., Li, J.-W., Li, K.S.-M., Tsai, N.C.-Y., Wang, S.-J., et al. (2021). Machine learning-based detection method for wafer test induced defects. IEEE Transactions on Semiconductor Manufacturing, 34(2), 161–167.
Choi, J., Kim, T., & Kim, C. (2019). Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 6830–6840).
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018). Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In 2018 IEEE/CVF conference on computer vision and pattern recognition (pp. 8789–8797).
Chuanfei, H., & Wang, Y. (2020). An efficient convolutional neural network model based on object-level attention mechanism for casting defect detection on radiography images. IEEE Transactions on Industrial Electronics, 67(12), 10922–10930.
DeVries, T., & Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552.
Dong, H., Kechen Song, Yu., He, J. X., Yan, Y., & Meng, Q. (2020). Pga-net: Pyramid feature fusion and global context attention network for automated surface defect detection. IEEE Transactions on Industrial Informatics, 16(12), 7448–7458.
Fan, R., Wang, H., Cai, P., Jin, W., Bocus, J., Qiao, L., & Liu, M. (2021). Learning collision-free space detection from stereo images: Homography matrix brings better data augmentation. IEEE/ASME Transactions on Mechatronics.
Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.-Y., Cubuk, E. D., Le, Q. V., & Zoph, B. (2021). Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2918–2928).
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial networks. arXiv preprint arXiv:1406.2661.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).
Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., & Lakshminarayanan, B. (2019). Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781.
Hsu, C.-Y., & Liu, W.-C. (2021). Multiple time-series convolutional neural network for fault detection and diagnosis and empirical study in semiconductor manufacturing. Journal of Intelligent Manufacturing, 32(3), 823–836.
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700–4708).
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In 2017 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 5967–5976).
Jain, S., Seth, G., Paruthi, A., Soni, U., & Kumar, G. (2020). Synthetic data augmentation for surface defect detection and classification using deep learning. Journal of Intelligent Manufacturing, 1–14.
Kim, Y., Cho, D., & Lee, J.-H. (2021). Wafer defect pattern classification with detecting out-of-distribution. Microelectronics Reliability, 122, 114157.
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25.
Le, X., Mei, J., Zhang, H., Zhou, B., & Xi, J. (2020). A learning-based approach for surface defect detection using small image datasets. Neurocomputing, 408, 112–120.
Li, W., Chen, J., Cao, J., Ma, C., Wang, J., Cui, X., & Chen, P. (2022). Eid-gan: Generative adversarial nets for extremely imbalanced data augmentation. IEEE Transactions on Industrial Informatics.
Lin, C.-T., Huang, S.-W., Yen-Yi, W., & Lai, S.-H. (2020). Gan-based day-to-night image style transfer for nighttime vehicle detection. IEEE Transactions on Intelligent Transportation Systems, 22(2), 951–963.
Lin, T. Y., Goyal, P., Girshick, R., He, K., & Doll, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980–2988).
Liu, C., Wang, K., Wang, Y., & Yuan, X. (2021). Learning deep multi-manifold structure feature representation for quality prediction with an industrial application. IEEE Transactions on Industrial Informatics.
Liu, J., Guo, F., Gao, H., Li, M., Zhang, Y., & Zhou, H. (2021). Defect detection of injection molding products on small datasets using transfer learning. Journal of Manufacturing Processes, 70, 400–413.
Liu, J., Wang, C., Hai, S., Bo, D., & Tao, D. (2019). Multistage gan for fabric defect detection. IEEE Transactions on Image Processing, 29, 3388–3400.
Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440).
Luo, M., Cao, J., Ma, X., Zhang, X., & He, R. (2021). Fa-gan: face augmentation gan for deformation-invariant face recognition. IEEE Transactions on Information Forensics and Security, 16, 2341–2355.
Müller, S. G., & Hutter, F. (2021). Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 774–782).
Naghizadeh, A., Xu, H., Mohamed, M., Metaxas, D. N., & Liu, D. (2021). Semantic aware data augmentation for cell nuclei microscopical images with artificial neural networks. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3952–3961).
Niu, S., Li, B., Wang, X., & Lin, H. (2020). Defect image sample generation with gan for improving defect recognition. IEEE Transactions on Automation Science and Engineering, 17(3), 1611–1622.
Pan, L., Rogulin, R., & Kondrashev, S. (2021). Artificial neural network for defect detection in ct images of wood. Computers and Electronics in Agriculture, 187, 106312.
Ren, X., Lin, W., Yang, X., Yu, X., & Gao, H. (2022). Data augmentation in defect detection of sanitary ceramics in small and non-iid datasets. IEEE Transactions on Neural Networks and Learning Systems.
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International conference on medical image computing and computer-assisted intervention (pp. 234–241). Springer.
Sassi, P., Tripicchio, P., & Avizzano, C. A. (2019). A smart monitoring system for automatic welding defect detection. IEEE Transactions on Industrial Electronics, 66(12), 9641–9650.
Schlosser, T., Friedrich, M., Beuth, F., & Kowerko, D. (2022). Improving automated visual fault inspection for semiconductor manufacturing using a hybrid multistage system of deep neural networks. Journal of Intelligent Manufacturing, 33(4), 1099–1123.
Singh, K. K., & Lee, Y. J. (2017). Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In 2017 IEEE international conference on computer vision (ICCV) (pp. 3544–3553). IEEE.
Su, B., Chen, H., Chen, P., Bian, G.-B., Liu, K., & Liu, W. (2021). Deep learning-based solar-cell manufacturing defect detection with complementary attention network. IEEE Transactions on Industrial Informatics, 17(6), 4084–4095.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818–2826).
Tabernik, D., Šela, S., Skvarč, J., & Skočaj, D. (2020). Segmentation-based deep-learning approach for surface-defect detection. Journal of Intelligent Manufacturing, 31(3), 759–776.
Tang, J., Zhou, H., Wang, T., Jin, Z., Wang, Y., & Wang, X. (2022). Cascaded foreign object detection in manufacturing processes using convolutional neural networks and synthetic data generation methodology. Journal of Intelligent Manufacturing 1–17.
Uricar, M., Sistu, G., Rashed, H., Vobecky, A., Kumar, V.R., Krizek, P., Burger, F., & Yogamani, S. (2021). Let’s get dirty: Gan based data augmentation for camera lens soiling detection in autonomous driving. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 766–775).
Wang, X., Man, Z., You, M., & Shen, C. (2017). Adversarial generation of training examples: applications to moving vehicle license plate recognition. arXiv preprint arXiv:1707.03124.
Xuan, Q., Chen, Z., Liu, Y., Huang, H., Bao, G., & Zhang, D. (2018). Multiview generative adversarial network and its application in pearl classification. IEEE Transactions on Industrial Electronics, 66(10), 8244–8252.
Yang, B., Liu, Z., Duan, G., & Tan, J. (2021). Mask2defect: A prior knowledge based data augmentation method for metal surface defect inspection. IEEE Transactions on Industrial Informatics.
Yun, J. P., Shin, W. C., Koo, G., Kim, M. S., Lee, C., & Lee, S. J. (2020). Automated defect inspection system for metal surfaces based on deep learning and data augmentation. Journal of Manufacturing Systems, 55, 317–324.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 6023–6032).
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.
Zhang, L., Wang, P., Li, H., Li, Z., Shen, C., & Zhang, Y. (2020). A robust attentional framework for license plate recognition in the wild. IEEE Transactions on Intelligent Transportation Systems, 22(11), 6967–6976.
Zheng, Z., Yang, X., Yu, Z., Zheng, L., Yang, Y., & Kautz, J. (2019). Joint discriminative and generative learning for person re-identification. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2138–2147).
Zheng, Z., Zheng, L., & Yang, Y. (2017). Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In Proceedings of the IEEE international conference on computer vision (pp. 3754–3762).
Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223–2232).
Acknowledgements
This work was supported in part by the National Key R &D Program of China under Grant 2018YFB1700500 and the Key Research and Development Program of HubeiChina (2020BAB106).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Niu, S., Peng, Y., Li, B. et al. A novel deep learning motivated data augmentation system based on defect segmentation requirements. J Intell Manuf 35, 687–701 (2024). https://doi.org/10.1007/s10845-022-02068-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-022-02068-y