Skip to main content
SpringerLink
Log in

IC solder joint inspection via generator-adversarial-network based template

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Automatic optical inspection is a vital part of the production process for solder joints appearance inspection in surface mounted technology assembling lines. However, IC solder joint inspection is a challenging problem because IC solder joints have extremely small sizes and no distinct appearance differences between qualified and unqualified ones. In this paper, we propose an IC solder joint inspection method via generator-adversarial-network based template. We are the first to introduce the GAN strategy into IC solder joint inspection. The method consists of GAN template generator training, offline statistical modelling and online real-time inspection. At the training stage, the GAN template generator is trained based on a designed GAN, which involves the feature maps in both of high-dimension and low-dimension spaces. Then, the binary difference image can be achieved by the input IC solder joint image and the corresponding GAN-based template. At the offline statistical modelling stage, to reduce the interferences, a pixel probability image is statistically modelled by the binary difference images corresponding to qualified IC solder joints. At the online real-time inspection stage, the potential defect pixels for the inspected IC solder joint can be shown in a defect salient image achieved by the multiplication of its corresponding binary difference image and the pixel probability image. Finally, we can accumulate the pixels in the defect salient image to distinguish the quality of the inspected IC solder joint. Experimental results show that the proposed method is superior to the state-of-the-art inspection methods with 0% omission rate and 0.15% error rate at a reasonable inspection speed of 4.32 ms per sample.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The processed data required to reproduce these findings cannot be shared at this time as the data also forms part of an ongoing study.

Notes

  1. The AOI image acquisition system is provided by EKTion (ShenZhen) Technology Co., LTD. Available at http://www.ekt-tech.com.

  2. Our models and code are publicly available at https://github.com/JiamingLi-zs/GAN-based-template.

References

  1. Huang, L., Shen, S., Xie, F., Zhao, J., Han, J., Feng, K.: A novel multi-pattern solder joint simultaneous segmentation algorithm for PCB selective packaging systems. Int. J. Pattern Recognit. Artif. Intell. 34, 1–21 (2019). https://doi.org/10.1142/S0218001420580057

    Article  Google Scholar 

  2. Chen, S.H., Perng, D.B.: Automatic optical inspection system for IC molding surface. J. Intell. Manuf. 27, 915–926 (2016). https://doi.org/10.1007/s10845-014-0924-5

    Article  Google Scholar 

  3. Su, L., Wang, L.Y., Li, K., Wu, J.J., Liao, G.L., Shi, T.L., Lin, T.Y.: Automated X-ray recognition of solder bump defects based on ensemble-ELM. Sci. China Technol. Sci. 62, 1512–1519 (2019). https://doi.org/10.1007/s11431-018-9324-3

    Article  Google Scholar 

  4. Song, J.D., Kim, Y.G., Park, T.H.: SMT defect classification by feature extraction region optimization and machine learning. Int. J. Adv. Manuf. Technol. 101, 1303–1313 (2019). https://doi.org/10.1007/s00170-018-3022-6

    Article  Google Scholar 

  5. Chang, P.C., Chen, L.Y., Fan, C.Y.: A case-based evolutionary model for defect classification of printed circuit board images. J. Intell. Manuf. 19, 203–214 (2008). https://doi.org/10.1007/s10845-008-0074-8

    Article  Google Scholar 

  6. Cai, N., Lin, J., Ye, Q., Wang, H., Weng, S., Ling, B.W.K.: A new IC solder joint inspection method for an automatic optical inspection system based on an improved visual background extraction algorithm. IEEE Trans. Compon. Packag. Manuf. Technol. 6, 161–172 (2016). https://doi.org/10.1109/TCPMT.2015.2501284

    Article  Google Scholar 

  7. Wu, F., Zhang, X.: Feature-extraction-based inspection algorithm for IC solder joints. IEEE Trans. Compon. Packag. Manuf. Technol. 1, 689–694 (2011). https://doi.org/10.1109/TCPMT.2011.2118208

    Article  Google Scholar 

  8. Wu, H., Zhang, X., Xie, H., Kuang, Y., Ouyang, G.: Classification of solder joint using feature selection based on bayes and support vector machine. IEEE Trans. Compon. Packag. Manuf. Technol. 3, 516–522 (2013). https://doi.org/10.1109/TCPMT.2012.2231902

    Article  Google Scholar 

  9. Xie, H., Zhang, X., Kuang, Y., Ouyang, G.: Solder joint inspection method for chip component using improved adaboost and decision tree. IEEE Trans. Compon. Packag. Manuf. Technol. 1, 2018–2027 (2011). https://doi.org/10.1109/TCPMT.2011.2168531

    Article  Google Scholar 

  10. Luo, B., Zhang, Y., Yu, G., Zhou, X.: ANN ensembles based machine vision inspection for solder joints. In: 2007 IEEE International Conference on Control & Automation, vol. 00, pp. 3111–3115 (2007). https://doi.org/10.1109/ICCA.2007.4376934

  11. Acciani, G., Brunetti, G., Fornarelli, G.: Application of neural networks in optical inspection and classification of solder joints in surface mount technology. IEEE Trans. Ind. Inform. 2, 200–209 (2006). https://doi.org/10.1109/TII.2006.877265

    Article  Google Scholar 

  12. Tae-Hyeon, K., Tai-Hoon, C., Moon, Y.S., Park, S.H.: Visual inspection system for the classification of solder joints. Pattern Recognit. 32, 565–575 (1999). https://doi.org/10.1016/s0031-3203(98)00103-4

    Article  Google Scholar 

  13. Ko, K.W., Cho, H.S.: Solder joints inspection using a neural network and fuzzy rule-based classification method. IEEE Trans. Electron. Packag. Manuf. 23, 78 (2000). https://doi.org/10.1109/6104.846932

    Article  Google Scholar 

  14. Lin, S.C., Chou, C.H., Su, C.H.: A development of visual inspection system for surface mounted devices on printed circuit board. In: IEEE Conference on Cybernetics & Intelligent Systems. pp. 2440–2445 (2007)

  15. Ong, T.Y., Samad, Z., Ratnam, M.M.: Solder joint inspection with multi-angle imaging and an artificial neural network. Int. J. Adv. Manuf. Technol. 38, 455–462 (2008). https://doi.org/10.1007/s00170-007-1117-6

    Article  Google Scholar 

  16. Luo, B., Zhang, Y., Yu, G., Zhou, X.: ANN ensembles based machine vision inspection for solder joints. In: 2007 IEEE International Conference on Control & Automation. ICCA, vol. 00, pp. 3111–3115 (2007). https://doi.org/10.1109/ICCA.2007.4376934

  17. Xie, H., Kuang, Y., Zhang, X.: A high speed AOI algorithm for chip component based on image difference. In: 2009 IEEE International Conference on Information and Automation(ICIA). pp. 969–974. IEEE (2009)

  18. Cai, N., Ye, Q., Liu, G., Wang, H., Yang, Z.: IC solder joint inspection based on the Gaussian mixture model. Solder. Surf. Mt. Technol. 28, 207–214 (2016). https://doi.org/10.1108/SSMT-03-2016-0005

    Article  Google Scholar 

  19. Wu, H., Xu, X.: Solder joint inspection using eigensolder features. Solder. Surf. Mt. Technol. 30, 227–232 (2018). https://doi.org/10.1108/SSMT-12-2017-0042

    Article  Google Scholar 

  20. Lin, H., Li, B., Wang, X., Shu, Y., Niu, S.: Automated defect inspection of LED chip using deep convolutional neural network. J. Intell. Manuf. 30, 2525–2534 (2019). https://doi.org/10.1007/s10845-018-1415-x

    Article  Google Scholar 

  21. Kaur, T., Gandhi, T.K.: Deep convolutional neural networks with transfer learning for automated brain image classification. Mach. Vis. Appl. 31, 20 (2020). https://doi.org/10.1007/s00138-020-01069-2

    Article  Google Scholar 

  22. Cai, N., Cen, G., Wu, J., Li, F., Wang, H., Chen, X.: SMT solder joint inspection via a novel cascaded convolutional neural network. IEEE Trans. Compon. Packag. Manuf. Technol. 8, 670–677 (2018). https://doi.org/10.1109/TCPMT.2018.2789453

    Article  Google Scholar 

  23. Dai, W., Mujeeb, A., Erdt, M., Sourin, A.: Towards automatic optical inspection of soldering defects. In: Proceedings of 2018 International Conference on Cyberworlds, pp. 375–382 (2018). https://doi.org/10.1109/CW.2018.00074

  24. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems. pp. 2672–2680 (2014)

  25. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: 4th International Conference on Learning Representations (ICLR). pp. 1–16 (2016)

  26. Wang, M., Chen, Z., Wu, Q.M.J., Jian, M.: Improved face super-resolution generative adversarial networks. Mach. Vis. Appl. 31, 22 (2020). https://doi.org/10.1007/s00138-020-01073-6

    Article  Google Scholar 

  27. Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: GANomaly: semi-supervised anomaly detection via adversarial training. In: Asian Conference on Computer Vision (ACCV), pp. 622–637. Springer, Cham (2019)

  28. Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: Skip-GANomaly: skip connected and adversarially trained encoder-decoder anomaly detection. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)

  29. Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Information Processing in Medical Imaging, pp. 146–157. Springer, Cham (2017)

  30. Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2813–2821 (2017)

  31. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems (2016)

Download references

Acknowledgements

This work was in part supported by the National Natural Science Foundation of China (Nos. 91648108 and 51875108), the Key Laboratory Construction Projects in Guangdong (No. 2017B030314178), the Research Fund for Colleges and Universities in Huizhou (No. 2019HZKY003), the Project of Jihua Laboratory (No.X190071UZ190) and the Science and Technology Program of Guangzhou, China (No. 201802020010).

Author information

Authors and Affiliations

  1. School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China

    Jiaming Li, Nian Cai, Zhuokun Mo & Guang Zhou

  2. Zhongshan Technical Secondary School, Zhongshan, 528400, China

    Jiaming Li

  3. Huizhou Guangdong University of Technology IoT Cooperative Innovation Institute Co., Ltd, Huizhou, 516025, China

    Nian Cai

  4. School of Electromechanical Engineering, Guangdong University of Technology, Guangzhou, 510006, China

    Han Wang

Authors
  1. Jiaming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nian Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhuokun Mo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Han Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors designed research, performed research, analysed data, and wrote the paper.

Corresponding author

Correspondence to Nian Cai.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Tables 2, 3 and 4.

Table 2 The parameters of the encoders GE and E2
Full size table
Table 3 The parameters of the decoder GD
Full size table
Table 4 The parameters of the discriminator D
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Cai, N., Mo, Z. et al. IC solder joint inspection via generator-adversarial-network based template. Machine Vision and Applications 32, 96 (2021). https://doi.org/10.1007/s00138-021-01218-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-021-01218-1

Keywords

Search

Navigation