Skip to main content
SpringerLink
Log in

Improving accuracy of automatic optical inspection with machine learning

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Electronic devices require the printed circuit board (PCB) to support the whole structure, but the assembly of PCBs suffers from welding problem of the electronic components such as surface mounted devices (SMDs) resistors. The automated optical inspection (AOI) machine, widely used in industrial production, can take the image of PCBs and examine the welding issue. However, the AOI machine could commit false negative errors and dedicated technicians have to be employed to pick out those misjudged PCBs. This paper proposes a machine learning based method to improve the accuracy of AOI. In particular, we propose an adjacent pixel RGB value based method to pre-process the image from the AOI machine and build a customized deep learning model to classify the image. We present a practical scheme including two machine learning procedures to mitigate AOI errors. We conduct experiments with the real dataset from a production line for three months, the experimental results show that our method can reduce the rate of misjudgment from 0.3%–0.5% to 0.02%–0.03%, which is meaningful for thousands of PCBs each containing thousands of electronic components in practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Blackwell G R. Electronic Systems Maintenance Handbook. 2nd ed. CRC Press, 2002

  2. Huang X, Zhu S, Huang X, Su B, Ou C, Zhou W. Detection of plated through hole defects in printed circuit board with X-ray. In: Proceedings of the 16th IEEE Intnational Conferenceon on Electronic Packaging Technology. 2015, 1296–1301

  3. Alaoui N E B, Tounsi P, Boyer A, Viard A. Detecting PCB assembly defects using infrared thermal signatures. In: Proceedings of International Conference “Mixed Design of Integrated Circuits and Systems”. 2019, 345–349

  4. Härter S, Klinger T, Franke J, Beer D. Comprehensive correlation of inline inspection data for the evaluation of defects in heterogeneous electronic assemblies. In: Proceedings of Pan Pacific Microelectronics Symposium. 2016, 1–6

  5. Wen K P, Wu W M, Huang C Y. Automatic optical inspection system and operating method thereof. U.S. Patent 10,438,340. 2019

  6. Runji J M, Lin C. Automatic optical inspection aided augmented reality-based PCBA inspection: a development. In: Proceedings of IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology. 2019, 667–671

  7. Wang W, Chen S, Chen L, Chang W. A machine vision based automatic optical inspection system for measuring drilling quality of printed circuit boards. IEEE Access, 2017, 5: 10817–10833

    Article  Google Scholar 

  8. Qiang G, Shanshan Z, Yang Z, Mao C. Detection method of PCB component based on automatic optical stitching algorithm. Circuit World, 2015, 41(4): 133–136

    Article  Google Scholar 

  9. Shorten C, Khoshgoftaar T M. A survey on image data augmentation for deep learning. Journal of Big Data, 2019, 6(1): 1–48

    Article  Google Scholar 

  10. Lengerich B, Xing E P, Caruana R. On dropout, overfitting, and interaction effects in deep neural networks. 2020, arXiv preprint arXiv: 2007.00823

  11. Liu X, Zhang J, Jiang S, Yang Y, Li K, Cao J, Liu J. Accurate localization of tagged objects using mobile RFID-augmented robots. IEEE Transactions on Mobile Computing, 2021, 20(4): 1273–1284

    Article  Google Scholar 

  12. Liu X, Chen S, Liu J, Qu W, Xiao F, Liu A X, Liu J. Fast and accurate detection of unknown tags for RFID systems-hash collisions are desirable. IEEE/ACM Transactions on Networking, 2020, 28(1):126–139

    Article  Google Scholar 

  13. Tong X, Liu K, Tian X, Fu L, Wang X. Fineloc: a fine-grained self-calibrating wireless indoor localization system. IEEE Transactions on Mobile Computing, 2018, 18(9): 2077–2090

    Article  Google Scholar 

  14. Du T B, Shen G H, Huang Z Q, Yu Y S, Wu D X. Automatic traceability link recovery via active learning. Frontiers of Information Technology & Electronic Engineering, 2020, 21(8): 1–9

    Article  Google Scholar 

  15. Huang J H, Di X G, Chen A Y. A novel convolutional neural network method for crowd counting. Frontiers of Information Technology & Electronic Engineering, 2020, 21(8): 1150–1160

    Article  Google Scholar 

  16. Alreshidi E. Smart sustainable agriculture (SSA) solution underpinned by internet of things (IoT) and artificial intelligence (AI). 2019, arXiv preprint arXiv: 1906.03106

  17. Tzafestas S G. Synergy of IoT and AI in modern society: the robotics and automation case. Robotics & Automation Engineering Journal, 2018, 31(5): 1–15

    Google Scholar 

  18. Xiao L, Wan X, Lu X, Zhang Y, Wu D. IoT security techniques based on machine learning: how do IoT devices use AI to enhance security? IEEE Signal Processing Magazine, 2018, 35(5): 41–49

    Article  Google Scholar 

  19. Meidan Y, Bohadana M, Shabtai A, Guarnizo J D, Ochoa M, Tippenhauer N O, Elovici Y. ProfilIoT: a machine learning approach for IoT device identification based on network traffic analysis. In: Proceedings of the Symposium on Applied Computing. 2017, 506–509

  20. Njima W, Ahriz I, Zayani R, Terre M, Bouallegueet R. Deep CNN for indoor localization in IoT-sensor systems. Journal of Sensors, 2019, 19(14): 3127–3132

    Article  Google Scholar 

  21. Canedo J, Skjellum A. Using machine learning to secure IoT systems. In: Proceedings of the 14th Annual Conference on Privacy, Security and Trust. 2016, 219–222

  22. Wang S, Tuor T, Salonidis T, Leung K K, Makaya C, He T, Chan K. When edge meets learning: adaptive control for resource-constrained distributed machine learning. In: Proceedings of the IEEE Conference on Computer Communications. 2018, 63–71

  23. Xiao L, Wan X, Lu X, Zhang Y, Wu D. IoT security techniques based on machine learning: how do IoT devices use AI to enhance security? IEEE Signal Processing Magazine, 2018, 35(5): 41–49

    Article  Google Scholar 

  24. Pramudita R, Hariadi F I. Development of techniques to determine object shifts for PCB board assembly automatic optical inspection. In: Proceedings of the International Symposium on Electronics and Smart Devices. 2018, 1–4

  25. Wu F, Li S, Zhao Y. A self-adaptive study method for multi-parameters thresholds in AOI system. In: Proceedings of the 11th World Congress on Intelligent Control and Automation. 2014, 5256–5259

  26. Jia X, Wang T, Li Y, Liu J, Zhang Y. AOI planning method based on genetic algorithm. In: Proceedings of International Conference on Mechatronics and Automation. 2019, 1801–1805

  27. Takacs T, Vajta L. Novel outlier filtering method for AOI image databases. IEEE Transactions on Components, Packaging and Manufacturing Technology, 2012, 2(4): 700–709

    Article  Google Scholar 

  28. Chaudhary V, Dave I R, Upla K P. Automatic visual inspection of printed circuit board for defect detection and classification. In: Proceedings of the International Conference on Wireless Communications, Signal Processing and Networking. 2017, 732–737

  29. Tsai J, Lin C, Chang C, Chou J. Optimized positional compensation parameters for exposure machine for flexible printed circuit board. IEEE Transactions on Industrial Informatics, 2015, 11(6): 1366–1377

    Article  Google Scholar 

  30. Mohammadi P, Wang Z J. Machine learning for quality prediction in abrasion-resistant material manufacturing process. In: Proceedings of the 2016 IEEE Canadian Conference on Electrical and Computer Engineering. 2016, 1–4

  31. Sartzetakis I, Christodoulopoulos K K, Varvarigos E M. Accurate quality of transmission estimation with machine learning. IEEE/OSA Journal of Optical Communications and Networking, 2019, 11(3): 140–150

    Article  Google Scholar 

  32. Von Enzberg S, Al-Hamadi A. A multiresolution approach to modelbased 3-D surface quality inspection. IEEE Transactions on Industrial Informatics, 2016, 12(4): 1498–1507

    Article  Google Scholar 

  33. Alonzo L M B, Chioson F B, Co H S, Bugtai N T, Baldovino R G. A machine learning approach for coconut sugar quality assessment and prediction. In: Proceedings of the 10th IEEE International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management. 2018, 1–4

  34. Sultanow E, Ullrich A, Konopik S, Vladova G. Machine learning based static code analysis for software quality assurance. In: Proceedings of the 13th International Conference on Digital Information Management. 2018, 156–161

  35. Li X, Zhang W, Ding Q, Li X. Diagnosing rotating machines with weakly supervised data using deep transfer learning. IEEE Transactions on Industrial Informatics, 2019, 16(3): 1688–1697

    Article  Google Scholar 

Download references

Acknowledgements

The work was supported by National Key Research and Development Program of China (2020YFB1708700), and the National Natural Science Foundation of China (Grant Nos. 61922055, 61872233, 61829201, 61532012, 61325012, 61428205).

Author information

Authors and Affiliations

  1. Department of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Xinyu Tong, Ziao Yu, Xiaohua Tian & Xinbing Wang

  2. Ambit Microsystems, Shanghai, 201600, China

    Houdong Ge

Authors
  1. Xinyu Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ziao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaohua Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Houdong Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xinbing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaohua Tian.

Additional information

Xinyu Tong received the BE and PhD degrees from the Department of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, China in 2015 and 2020, respectively. He holds a post-doctoral position at the College of Intelligence and Computing, Tianjin University, China. His research interests include wireless sensor network and wireless localization.

Ziao Yu received the BE degree in Department of Electronic Information and Electrical Engineering from Shanghai Jiao Tong University, China in 2019. He plans to go abroad to pursue a master’s degree. His research interests include machine learning and data science.

Xiaohua Tian received his BE and ME degrees in communication engineering from Northwestern Polytechnical University, China in 2003 and 2006, respectively. He received the PhD degree in the Department Electrical and Computer Engineering (ECE), Illinois Institute of Technology (IIT), Chicago in 2010. Currently, he is a professor in the Department of Electronic Engineering, Shanghai Jiao Tong University, China. He serves as the scanning the literature column editor of IEEE Network Magazine (2016–) and the guest editor of IEEE Internet of Things Journal (2018–2019). He also serves as the Vice Program co-chair of ACM Turing Celebration Conference (TURC) 2019, TPC member for IEEE INFOCOM 2014–2018, 2020, symposium co-chair of IEEE/CIC ICCC 2015, 2019.

Houdong Ge is the factory manager of Ambit Microsystems (Shanghai) Ltd, China. He has more than 15 years experience in manufacturing consumer electronics. He is proficient in manufacture process development, automation and lean production. Moreover, he is interested in surveying and implementing new technology to production line.

Xinbing Wang received the BS degree (with hons.) from the Department of Automation, Shanghai Jiaotong University, China in 1998, and the MS degree from the Department of Computer Science and Technology, Tsinghua University, China in 2001. He received the PhD degree, major in the Department of electrical and Computer Engineering, North Carolina State University, USA in 2006. Currently, he is a professor in the Department of Electronic Engineering, Shanghai Jiaotong University, China, Dr. Wang has been an associate editor for IEEE/ACM Transactions on Networking and IEEE Transactions on Mobile Computing, and the member of the Technical Program Committees of several conferences including ACM MobiCom 2012, 2018–2021, ACM Sigmetrics 2021, ACM MobiHoc 2012–2019, IEEE INFOCOM 2009–2020.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tong, X., Yu, Z., Tian, X. et al. Improving accuracy of automatic optical inspection with machine learning. Front. Comput. Sci. 16, 161310 (2022). https://doi.org/10.1007/s11704-021-0244-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-021-0244-9

Keywords

Search

Navigation