Scale insensitive and focus driven mobile screen defect detection in industry
Introduction
With the rapid development of mobile communication technology, the human has entered the mobile Internet era. Smartphones gradually become an indispensable part of people’s lives. The function of smartphones has already surpassed the field of text communication and telephone communication. Instead, smartphones have become an important way for people to socialize and get information. The production quantity of global smartphones is raising year by year, making an urgent need of improving quality and ensuring production efficiency for smartphone manufacturers.
As an important component of smartphones, mobile phone screen plays a pivotal role in the user experience. The quality testing of the mobile phone screen is thus a crucial link in production. Mobile phone screen has developed from the black and white screen to the color screen, and then to the current high-definition display. Now the mobile phone screen can display a variety of complex images vividly with rich layering. At the same time, the production process of the mobile phone screen is more and harsher. It is susceptible to the production environment and other factors, resulting in various types of defects, such as dead pixels, light leakage, color difference and so on, as shown in Fig. 1. Considering the yield rate of the mobile phone screen is relatively low, mobile phone manufacturers have to take some means to check the quality of the phone screen to prevent defective mobile phone screen products sold into the market.
The traditional way to detect mobile phone screen defects is to arrange inspectors on the production line, where they use the naked eyes to detect the existence of defects on the phone screen in order. However, with the growing demand for mobile phone screen market, using the manual detection of mobile phone screen defects reveals many drawbacks:
- •
Both detection efficiency and speed are slow. There are many small defects on the phone screen that are difficult to detect through the human eyes, resulting in low efficiency of manual detection and the production line.
- •
A lack of criteria. Manual detection is mainly judged by human visual senses, which varies from different individuals. As a consequence, it is difficult to summarize a unified criterion. Different people may hold different test results on the same mobile phone screen product.
- •
Labor cost is relatively high. Employing a large number of inspectors to detect defects on mobile phone screen will greatly increase the enterprise labor cost and reducing the market competitiveness of industries.
Because of these shortcomings, traditional manual detection methods are unable to adapt to current industrial production requirements of efficiency and accuracy. In this paper, a high-resolution industrial camera is adopted to collect mobile phone screen images. To solve the above shortcomings, we proposed an end-to-end automated framework integrated with two deep networks for mobile phone screen defect detection. The first network (MSDDN) aims at parallel detecting defects under multiple scales. The second one (SCN) is designed for predicting the area with the most probability of containing defeats and eliminating the environmental factors on capturing. Experimental results illustrate that with a combination of the proposed two networks, our framework can achieve robust performance and efficiency.
Section snippets
Filter based methods
In the mobile phone screen defect detection, the relatively obvious defects can be recognized with edge detection. Common edge detection operators include Roberts operator, Sobel operator, Laplacian operator and Canny operator and so on [1], [2], [3]. Roberts operator is one of the simplest edge detection algorithms. However, it is sensitive to the noise in the image and apt to produce isolated points in the calculation results. Sobel operator has a smooth effect on the noise and can produce
Scale factor in SDD
The scale varies greatly among different mobile phone screen defects. Thus, defects should been observed at different scales to get better views from human vision systems. As shown in Fig. 2, there are two defects on the left, referring as Da and Db. We focus on the patches around them and blow up the image at different levels, from 1.5 × to 3 × . It can be discovered that Da is better viewed at 1.5 × and Db at 2.5 × .
The convolution operator can be regarded as an information centralizing
SDD criterion
MSDDN can decide whether the input image contains defects. However, the whole screen image can be of high resolutions, raising the training difficulty. Besides, the training process of MSDDN may suffer from data shortage situation. A feasible solution to this problem is cropping image patches from the original screen image. Each patch can be calculated an overlap area with defects. When the area is larger than a threshold, the patch is then marked as a negative sample. In this way, the training
Network compression
Deep network models have achieved superior performance over the traditional methods on the cost of large-scale parameters and deeper structures. Even though with GPU employed, the time cost for the prediction process is still unacceptable to meet the demand in many application scenarios. Meanwhile, the parameters of the model also take up a lot of memory space.
The proposed MSDDN is mainly composed of convolutional layers, nonlinear activation layers, and joint layers. Although the convolution
Dataset
We collect a 3000 image dataset collected in various product batches and screen models, where each defect in an image is annotated with a bounding box by experienced inspectors. Angular point detection is firstly performed on each original screen image to decide the bounds for the screen, as shown in Fig. 6. To obtain the screen images, an affine transformation is subsequently applied to eliminate lens distortion. We then crop from these images with random positions to obtain multiple screen
Conclusion
In this work, we present an end-to-end mobile screen defect detection framework, which is composed of a scale insensitive MSDDN network and a self-comparison driven SCN network. MSDDN integrates multiple subnetworks to deal with various defect scales. SCN investigates the differences between areas cropped from the same image to decide the one with most probability to contain defects as a focus. As we set an image area with much larger size than an image patch, combining MSDDN with SCN can
Acknowledgment
This work is supported by National Key Research and Development Program (2016YFB1200203), National Natural Science Foundation of China (61572428,U1509206), Fundamental Research Funds for the Central Universities (2017FZA5014).
Jie Lei is currently a Ph.D. student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include computer vision, robotic vision and machine learning.
References (30)
A survey of edge detection techniques
Comput. Graph. Image Process.
(1975)- et al.
A study of edge detection algorithms
Comput. Graph. Image Process.
(1982) - et al.
Convolutional neural networks for hyperspectral image classification
Neurocomputing
(2017) - et al.
Remote sensing image classification based on neural network ensemble algorithm
Neurocomputing
(2012) - et al.
Image classification based on effective extreme learning machine
Neurocomputing
(2013) - et al.
Remaining useful life estimation of engineered systems using vanilla LSTM neural networks
Neurocomputing
(2018) - et al.
Encoding subcomponents in cooperative co-evolutionary recurrent neural networks
Neurocomputing
(2011) A computational approach to edge detection
IEEE Trans. Pattern Anal. Mach. Intell.
(1986)- et al.
Automatic defect inspection for LCDs using singular value decomposition
Int. J. Adv. Manuf. Technol.
(2005) Fourier Analysis and its Applications
(1992)
A signal processing approach to fourier analysis of ranking data: the importance of phase
IEEE Trans. Signal Process.
The importance of phase in signals
Proc. IEEE
Discriminative multi-view interactive image re-ranking
IEEE Trans. Image Process.
DCT regularized extreme visual recovery
IEEE Trans. Image Process.
Tag disentangled generative adversarial networks for object image re-rendering
Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI
Cited by (0)
Jie Lei is currently a Ph.D. student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include computer vision, robotic vision and machine learning.
Xin Gao is currently a master student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include machine learning and computer vision.
Zunlei Feng is currently a Ph.D. student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include machine learning, computer vision and color management.
Huamou Qiu is currently a master student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include machine learning, computer vision and industrial vision.
Mingli Song (M’06-SM’13) is a Professor in Microsoft Visual Perception Laboratory, Zhejiang University. He received the Ph.D. degree in Computer Science from Zhejiang University, China, in 2006. He was awarded Microsoft Research Fellowship in 2004. His research interests include face modeling and facial expression analysis.