Elsevier

Neurocomputing

Volume 294, 14 June 2018, Pages 72-81
Neurocomputing

Scale insensitive and focus driven mobile screen defect detection in industry

https://doi.org/10.1016/j.neucom.2018.03.013Get rights and content

Abstract

With the wide-spread of smartphones, mobile phone screen has become an important IO device in HCI and its quality is of great matter in interaction. Traditional defect detection process involves heavy labor cost or relies on unstable low-level features and suffers from both scale and model sensitive problems. Screen defect varies in size, shape, intensity and is hard to be described. Efficient and accurate detection system remains an urgent need in mobile phone screen manufacturing.

In this paper, we propose an end-to-end screen defect detection framework. A defect detection network with merging and splitting strategies (MSDDN) to deal with multiple size and shape variations of defect image patches is firstly designed. After training, feature maps of the last layer before the output of MSDDN can be regarded as good representations of a screen image patch. These feature maps are concatenated into a unified feature vector. We then train a recurrent neural network (SCN) to decide which input screen image patch in a sequence is the most likely to contain defects, where the patches are cropped from the same image, and the feature maps are used as input. As SCN emphasizes on the comparison of image patches from one image, it is less sensitive to different screen batches. The patch with the highest probability of containing defects, or called focus area, is further processed with a sliding window to fill in the MSDDN to produce the final results. Finally, to improve the efficiency of the calculation process to fulfill the real industrial demand, we perform both filter selection and weight quantization on the weights in MSDDN under the purpose of building a low-precision version network without great loss in accuracy (MSDDN-l). Experimental results show MSDDN can better handle the defect variations than traditional models and general purpose convolutional neural networks. Meanwhile, SCN can accurately predict the focus area, and MSDDN-l can greatly improve the efficiency.

Introduction

With the rapid development of mobile communication technology, the human has entered the mobile Internet era. Smartphones gradually become an indispensable part of people’s lives. The function of smartphones has already surpassed the field of text communication and telephone communication. Instead, smartphones have become an important way for people to socialize and get information. The production quantity of global smartphones is raising year by year, making an urgent need of improving quality and ensuring production efficiency for smartphone manufacturers.

As an important component of smartphones, mobile phone screen plays a pivotal role in the user experience. The quality testing of the mobile phone screen is thus a crucial link in production. Mobile phone screen has developed from the black and white screen to the color screen, and then to the current high-definition display. Now the mobile phone screen can display a variety of complex images vividly with rich layering. At the same time, the production process of the mobile phone screen is more and harsher. It is susceptible to the production environment and other factors, resulting in various types of defects, such as dead pixels, light leakage, color difference and so on, as shown in Fig. 1. Considering the yield rate of the mobile phone screen is relatively low, mobile phone manufacturers have to take some means to check the quality of the phone screen to prevent defective mobile phone screen products sold into the market.

The traditional way to detect mobile phone screen defects is to arrange inspectors on the production line, where they use the naked eyes to detect the existence of defects on the phone screen in order. However, with the growing demand for mobile phone screen market, using the manual detection of mobile phone screen defects reveals many drawbacks:

  • Both detection efficiency and speed are slow. There are many small defects on the phone screen that are difficult to detect through the human eyes, resulting in low efficiency of manual detection and the production line.

  • A lack of criteria. Manual detection is mainly judged by human visual senses, which varies from different individuals. As a consequence, it is difficult to summarize a unified criterion. Different people may hold different test results on the same mobile phone screen product.

  • Labor cost is relatively high. Employing a large number of inspectors to detect defects on mobile phone screen will greatly increase the enterprise labor cost and reducing the market competitiveness of industries.

Because of these shortcomings, traditional manual detection methods are unable to adapt to current industrial production requirements of efficiency and accuracy. In this paper, a high-resolution industrial camera is adopted to collect mobile phone screen images. To solve the above shortcomings, we proposed an end-to-end automated framework integrated with two deep networks for mobile phone screen defect detection. The first network (MSDDN) aims at parallel detecting defects under multiple scales. The second one (SCN) is designed for predicting the area with the most probability of containing defeats and eliminating the environmental factors on capturing. Experimental results illustrate that with a combination of the proposed two networks, our framework can achieve robust performance and efficiency.

Section snippets

Filter based methods

In the mobile phone screen defect detection, the relatively obvious defects can be recognized with edge detection. Common edge detection operators include Roberts operator, Sobel operator, Laplacian operator and Canny operator and so on [1], [2], [3]. Roberts operator is one of the simplest edge detection algorithms. However, it is sensitive to the noise in the image and apt to produce isolated points in the calculation results. Sobel operator has a smooth effect on the noise and can produce

Scale factor in SDD

The scale varies greatly among different mobile phone screen defects. Thus, defects should been observed at different scales to get better views from human vision systems. As shown in Fig. 2, there are two defects on the left, referring as Da and Db. We focus on the patches around them and blow up the image at different levels, from 1.5 ×  to 3 × . It can be discovered that Da is better viewed at 1.5 ×  and Db at 2.5 × .

The convolution operator can be regarded as an information centralizing

SDD criterion

MSDDN can decide whether the input image contains defects. However, the whole screen image can be of high resolutions, raising the training difficulty. Besides, the training process of MSDDN may suffer from data shortage situation. A feasible solution to this problem is cropping image patches from the original screen image. Each patch can be calculated an overlap area with defects. When the area is larger than a threshold, the patch is then marked as a negative sample. In this way, the training

Network compression

Deep network models have achieved superior performance over the traditional methods on the cost of large-scale parameters and deeper structures. Even though with GPU employed, the time cost for the prediction process is still unacceptable to meet the demand in many application scenarios. Meanwhile, the parameters of the model also take up a lot of memory space.

The proposed MSDDN is mainly composed of convolutional layers, nonlinear activation layers, and joint layers. Although the convolution

Dataset

We collect a 3000 image dataset collected in various product batches and screen models, where each defect in an image is annotated with a bounding box by experienced inspectors. Angular point detection is firstly performed on each original screen image to decide the bounds for the screen, as shown in Fig. 6. To obtain the screen images, an affine transformation is subsequently applied to eliminate lens distortion. We then crop from these images with random positions to obtain multiple screen

Conclusion

In this work, we present an end-to-end mobile screen defect detection framework, which is composed of a scale insensitive MSDDN network and a self-comparison driven SCN network. MSDDN integrates multiple subnetworks to deal with various defect scales. SCN investigates the differences between areas cropped from the same image to decide the one with most probability to contain defects as a focus. As we set an image area with much larger size than an image patch, combining MSDDN with SCN can

Acknowledgment

This work is supported by National Key Research and Development Program (2016YFB1200203), National Natural Science Foundation of China (61572428,U1509206), Fundamental Research Funds for the Central Universities (2017FZA5014).

Jie Lei is currently a Ph.D. student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include computer vision, robotic vision and machine learning.

References (30)

  • R. Kakarala

    A signal processing approach to fourier analysis of ranking data: the importance of phase

    IEEE Trans. Signal Process.

    (2011)
  • A.V. Oppenheim et al.

    The importance of phase in signals

    Proc. IEEE

    (1981)
  • J. Li et al.

    Discriminative multi-view interactive image re-ranking

    IEEE Trans. Image Process.

    (2017)
  • Y. Wang et al.

    DCT regularized extreme visual recovery

    IEEE Trans. Image Process.

    (2017)
  • C. Wang et al.

    Tag disentangled generative adversarial networks for object image re-rendering

    Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI

    (2017)
  • Cited by (0)

    Jie Lei is currently a Ph.D. student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include computer vision, robotic vision and machine learning.

    Xin Gao is currently a master student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include machine learning and computer vision.

    Zunlei Feng is currently a Ph.D. student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include machine learning, computer vision and color management.

    Huamou Qiu is currently a master student in the College of Computer Science, Zhejiang University, Zhejiang, China. His research interests include machine learning, computer vision and industrial vision.

    Mingli Song (M’06-SM’13) is a Professor in Microsoft Visual Perception Laboratory, Zhejiang University. He received the Ph.D. degree in Computer Science from Zhejiang University, China, in 2006. He was awarded Microsoft Research Fellowship in 2004. His research interests include face modeling and facial expression analysis.

    View full text