Abstract
Foreign object detection in manufacturing processes based on machine vision remains a challenge. The vastly different foreign objects and the complex background, as well as the scarcity of images with foreign objects constrain the application of traditional and deep learning methods, respectively. This paper discusses a novel method for intelligent foreign object detection and automatic data generation. A cascaded convolutional neural network to detect foreign objects on the surface of the tobacco pack is proposed. The cascaded network transforms the inspection into a two-stage YOLO based object detection, consisting of the tobacco pack localization and the foreign object detection. To address the scarcity of images with foreign objects, several data augmentation methods are introduced to avoid overfitting. Furthermore, a data generation methodology based on homography transformation and image fusion is developed to generate synthetic images with foreign objects. Models trained using synthetic images generated by this method show superior performance, which presents a viable approach to detecting newly introduced foreign objects. Extensive experimental results and comparisons on the tobacco pack foreign object dataset with several state-of-the-art methods demonstrate the effectiveness and superiority of the proposed method. The proposed cascaded foreign object detection network and synthetic data generation methodology have the potential for widespread applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bochkovskiy, A., Wang, C. Y., & Liao, H. Y. M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv:2004.10934
Chen, J., Liu, Z., Wang, H., Nunez, A., & Han, Z. (2018). Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 67(2), 257–269. https://doi.org/10.1109/TIM.2017.2775345.
Condorí, M. , Albesa, F., Altobelli, F., Duran, G., & Sorrentino, C. (2020). Image processing for monitoring of the cured tobacco process in a bulk-curing Dstove. Computers and Electronics in Agriculture. https://doi.org/10.1016/j.compag.2019.105113
Dhillon, A., & Verma, G. K. (2020). Convolutional neural network: A review of models, methodologies and applications to object detection. Progress in Artificial Intelligence, 9(2), 85–112. https://doi.org/10.1007/s13748-019-00203-0.
Giesko, T., Mazurkiewicz, A., Garbacz, P., Czajka, P., Sikora, L., & Dobrowolski, J. (2018). Innovative optomechatronic technologies in the tobacco industry. Journal of Machine Construction and Maintenance - Problemy Eksploatacji 323–332.
Guo, R., Qu, L., Niu, D., Li, Z., & Yue, J. (2021). LeafMask: Towards greater accuracy on leaf segmentation. arXiv:2108.03568
Haralick, R. M. (1980). Using perspective transformations in scene analysis. Computer Graphics and Image Processing, 13(3), 191–221. https://doi.org/10.1016/0146-664X(80)90046-5.
Harjoko, A., Prahara, A., Supardi, T. W., Candradewi, I., Pulungan, R., & Hartati, S. (2019). Image processing approach for grading tobacco leaf based on color and quality. International Journal on Smart Sensing and Intelligent Systems, 12(1), 1–10.
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9), 1904–1916. https://doi.org/10.1109/TPAMI.2015.2389824.
Huang, F., Wang, B., Li, Q., & Zou, J. (2021). Texture surface defect detection of plastic relays with an enhanced feature pyramid network. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-021-01864-2.
Jin, Z., Zhong, F., Zhang, Q., Wang, W., & Wang, X. (2021). Visual detection of tobacco packaging film based on apparent features. International Journal of Advanced Robotic Systems. https://doi.org/10.1177/17298814211024839.
Kim, T. S., Lee, J. W., Lee, W. K., & Sohn, S. Y. (2021). Novel method for detection of mixed-type defect patterns in wafer maps based on a single shot detector algorithm. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-021-01755-6.
Lee, J. H., Lee, S., Zhang, G., Lim, J., Chung, W. K., & Suh, I. H. (2014). Outdoor place recognition in urban environments using straight lines. In IEEE international conference on robotics and automation (pp. 5550–5557). https://doi.org/10.1109/ICRA.2014.6907675
Li, G., Zhen, H., Jiao, F., Hao, T., Wang, D., & Ni, K. (2021). Research on tobacco leaf grading algorithm based on transfer learning. In IEEE international conference on artificial intelligence and computer applications (pp. 32–35). https://doi.org/10.1109/icaica52286.2021.9497953
Lin, T. Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. In IEEE conference on computer vision and pattern recognition (pp. 2117–2125). https://doi.org/10.1109/cvpr.2017.106
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single shot multibox detector. In European conference on computer vision (pp. 21–37). https://doi.org/10.1007/978-3-319-46448-0_2
Liu, X., Yang, T., & Li, J. (2018). Real-time ground vehicle detection in aerial infrared imagery based on convolutional neural network. Electronics, 7(6), 78. https://doi.org/10.3390/electronics7060078.
Luo, H., & Zhang, C. (2018). Features representation for flue-cured tobacco grading based on transfer learning to hard sample. In IEEE international conference on signal processing (pp. 591–595). https://doi.org/10.1109/icsp.2018.8652385
Lv, Z., Wang, W., Xu, Z., Zhang, K., & Lv, H. (2021). Cascade network for detection of coal and gangue in the production context. Powder Technology, 377, 361–371. https://doi.org/10.1016/j.powtec.2020.08.088.
MacCormac, J. (1993). On-line image processing for tobacco grading in Zimbabwe. In IEEE international symposium on industrial electronics conference proceedings (pp. 327–331). https://doi.org/10.1109/isie.1993.268786
Marzan, C. S., & Ruiz, C. R., Jr. (2019). Automated tobacco grading using image processing techniques and a convolutional neural network. International Journal of Machine Learning and Computing, 9(6), 807–813. https://doi.org/10.18178/ijmlc.2019.9.6.877.
Mi, C., Chen, K., & Zhang, Z. (2020). Research on tobacco foreign body detection device based on machine vision research on tobacco foreign body detection device based on machine vision. Transactions of the Institute of Measurement and Control, 42(15), 2857–2871. https://doi.org/10.1177/0142331220929816.
Pérez, P., Gangnet, M., & Blake, A. (2003). Poisson image editing. ACM SIGGRAPH 2003 Papers (pp. 313–318). https://doi.org/10.1145/1201775.882269
Redmon, J., & Farhadi, A. (2018). YOLOv3: An incremental improvement. arXiv:1804.02767
Rehman, T. U., Mahmud, M. S., Chang, Y. K., Jin, J., & Shin, J. (2019). Current and future applications of statistical machine learning algorithms for agricultural machine vision systems. Computers and Electronics in Agriculture, 156, 585–605. https://doi.org/10.1016/j.compag.2018.12.006.
Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In International conference on medical image computing and computer-assisted intervention (pp. 234–241). https://doi.org/10.1007/978-3-319-24574-4_28
Shen, L., You, L., Peng, B., & Zhang, C. (2021). Group multi-scale attention pyramid network for traffic sign detection. Neurocomputing, 452, 1–14. https://doi.org/10.1016/j.neucom.2021.04.083.
Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 60. https://doi.org/10.1186/s40537-019-0197-0.
Tao, X., Zhang, D., Wang, Z., Liu, X., Zhang, H., & Xu, D. (2020). Detection of power line insulator defects using aerial images analyzed with convolutional neural networks. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 50(4), 1486–1498. https://doi.org/10.1109/TSMC.2018.2871750.
Thirthe Gowda, M.T., & Chandrika, J. (2021). Analysis and classification of ripped tobacco leaves using machine learning techniques. In Evolutionary computing and mobile sustainable networks (pp. 171–180). https://doi.org/10.1007/978-981-15-5258-8_18
Wang, L., Cheng, B., Li, Z., Liu, T., & Li, J. (2017). Intelligent tobacco flue-curing method based on leaf texture feature analysis. Optik, 150, 117–130. https://doi.org/10.1016/j.ijleo.2017.09.088.
Wu, J., & Yang, S. X. (2021). Modeling of the bulk tobacco flue-curing process using a deep learning-based method. IEEE Access, 9, 140424–140436. https://doi.org/10.1109/ACCESS.2021.3119544.
Zhang, J., Sokhansanj, S., Wu, S., Fang, R., & Yang, W. (1997). A trainable grading system for tobacco leaves. Computers and Electronics in Agriculture, 16(3), 231–244. https://doi.org/10.1016/s0168-1699(96)00040-3.
Zhang, J., Xie, Z., Sun, J., Zou, X., & Wang, J. (2020). A cascaded R-CNN with multiscale attention and imbalanced samples for traffic sign detection. IEEE Access, 8, 29742–29754. https://doi.org/10.1109/ACCESS.2020.2972338.
Zhang, Z., He, T., Zhang, H., Zhang, Z., Xie, J., & Li, M. (2019). Bag of freebies for training object detection neural networks. arXiv:1902.04103
Zheng, Z., Wang, P., Liu, W., Li, J., Ye, R., & Ren, D. (2019). Distance-IoU Loss: Faster and better learning for bounding box regression. arXiv:1911.08287
Zhou, X., Koltun, V., & Krähenbühl, P. (2021). Probabilistic two-stage detection. arXiv:2103.07461
Zoph, B., Cubuk, E. D., Ghiasi, G., Lin, T. Y., Shlens, J., & Le, Q. V. (2020). Learning data augmentation strategies for object detection. European Conference on Computer Vision. https://doi.org/10.1007/978-3-030-58583-9_34.
Acknowledgements
This work was supported by science and technology project of China Tobacco Zhejiang Industrial Co., Ltd. under Grant No. ZJZY2020E003.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tang, J., Zhou, H., Wang, T. et al. Cascaded foreign object detection in manufacturing processes using convolutional neural networks and synthetic data generation methodology. J Intell Manuf 34, 2925–2941 (2023). https://doi.org/10.1007/s10845-022-01976-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-022-01976-3