Skip to main content
Springer Nature Link
Log in

A Comparison of Model Confidence Metrics on Visual Manufacturing Quality Data

  • Conference paper
  • First Online:
Computer Vision and Machine Intelligence

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 586))

  • 781 Accesses

Abstract

After ground-breaking achievements through the application of modern deep learning, there is a considerable push towards using machine learning systems for visual inspection tasks part of most industrial manufacturing processes. But whilst there exist a lot of successful proof-of-concept implementations, productive use proves problematic. Whilst missing interpretability is one concern, the constant presence of data drift is another. Changes in pre-materials or process and degradation of sensors or product redesigns impose constant change towards statically trained machine learning models. To handle these kind of changes, a measurement of system confidence is needed. Since pure model output probabilities often lack in this concern better solutions are required. In this work, we compare and contrast several pre-existing methods used to describe model confidence. In contrast to previous works, they are evaluated on a large set of real-world manufacturing data. It is shown that utilizing an approach based on auto-encoder reconstruction error proves to be most promising in all scenarios tested.

This work was supported by OSRAM Automotive.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abati, D., Porrello, A., Calderara, S., Cucchiara, R.: Latent space autoregression for novelty detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  2. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in AI safety (2016). https://doi.org/10.48550/ARXIV.1606.06565, https://arxiv.org/abs/1606.06565

  3. An, J., Cho, S.: Variational autoencoder based anomaly detection using reconstruction probability. Spec. Lect. IE 2(1), 1–18 (2015)

    Google Scholar 

  4. Barros, R.S.M., Santos, S.G.T.C.: A large-scale comparison of concept drift detectors. Inf. Sci. 451–452, 348–370 (2018). https://doi.org/10.1016/j.ins.2018.04.014, http://www.sciencedirect.com/science/article/pii/S0020025518302743

  5. Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy, pp. 39–57. IEEE (2017)

    Google Scholar 

  6. Chalapathy, R., Chawla, S.: Deep learning for anomaly detection: a survey. arXiv preprint arXiv:1901.03407 (2019)

  7. Du, X., Wang, Z., Cai, M., Li, Y.: VOS: learning what you don’t know by virtual outlier synthesis. CoRR abs/2202.01197, https://arxiv.org/abs/2202.01197 (2022)

  8. Forza, C.: Work organization in lean production and traditional plants: what are the differences? Int. J. Oper. Prod. Manage. (1996)

    Google Scholar 

  9. Gong, D., Liu, L., Le, V., Saha, B., Mansour, M.R., Venkatesh, S., Hengel, A.V.D.: Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  10. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)

  11. Grigorescu, S., Trasnea, B., Cocias, T., Macesanu, G.: A survey of deep learning techniques for autonomous driving. J. Field Rob. 37(3), 362–386 (2020)

    Article  Google Scholar 

  12. Guo, X., Liu, X., Zhu, E., Yin, J.: Deep clustering with convolutional autoencoders. In: International Conference on Neural Information Processing, pp. 373–382. Springer (2017)

    Google Scholar 

  13. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks (2016). https://doi.org/10.48550/ARXIV.1603.05027, https://arxiv.org/abs/1603.05027

  14. Hein, M., Andriushchenko, M., Bitterwolf, J.: Why Relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41–50 (2019)

    Google Scholar 

  15. Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks (2018)

    Google Scholar 

  16. Jiang, H., Kim, B., Guan, M.Y., Gupta, M.R.: To trust or not to trust a classifier. In: NeurIPS, pp. 5546–5557 (2018)

    Google Scholar 

  17. Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A Math. Phys. Eng. Sci. 374(2065), 20150202 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  18. Jordaney, R., Sharad, K., Dash, S.K., Wang, Z., Papini, D., Nouretdinov, I., Cavallaro, L.: Transcend: detecting concept drift in malware classification models. In: 26th USENIX Security Symposium (USENIX Security 17), pp. 625–642. USENIX Association, Vancouver, BC (2017). https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/jordaney

  19. Laugel, T., Lesot, M.J., Marsala, C., Renard, X., Detyniecki, M.: The dangers of post-hoc interpretability: unjustified counterfactual explanations. arXiv preprint arXiv:1907.09294 (2019)

  20. Lee, K.B., Cheon, S., Kim, C.O.: A convolutional neural network for fault classification and diagnosis in semiconductor manufacturing processes. IEEE Trans. Semicond. Manuf. 30(2), 135–142 (2017)

    Article  Google Scholar 

  21. Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  22. Li, D., Liang, L.Q., Zhang, W.J.: Defect inspection and extraction of the mobile phone cover glass based on the principal components analysis. Int. J. Adv. Manuf. Technol. 73(9), 1605–1614 (2014)

    Article  Google Scholar 

  23. Lin, C.C., Deng, D.J., Kuo, C.H., Chen, L.: Concept drift detection and adaption in big imbalance industrial IoT data using an ensemble learning method of offline classifiers. IEEE Access 7, 56198–56207 (2019)

    Article  Google Scholar 

  24. Malhi, A., Gao, R.X.: PCA-based feature selection scheme for machine defect classification. IEEE Trans. Instrum. Measur. 53(6), 1517–1525 (2004)

    Article  Google Scholar 

  25. Mandelbaum, A., Weinshall, D.: Distance-based confidence score for neural network classifiers. arXiv preprint arXiv:1709.09844 (2017)

  26. Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  27. Ozdemir, R., Koc, M.: A quality control application on a smart factory prototype using deep learning methods. In: 2019 IEEE 14th International Conference on Computer Sciences and Information Technologies (CSIT), vol. 1, pp. 46–49. IEEE (2019)

    Google Scholar 

  28. Papernot, N., McDaniel, P.: Deep k-nearest neighbors: towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765 (2018)

  29. Piccialli, F., Di Somma, V., Giampaolo, F., Cuomo, S., Fortino, G.: A survey on deep learning in medicine: why, how and when? Inf. Fusion 66, 111–137 (2021)

    Article  Google Scholar 

  30. Reis, D.M.d., Flach, P., Matwin, S., Batista, G.: Fast unsupervised online drift detection using incremental Kolmogorov-Smirnov test. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1545–1554 (2016)

    Google Scholar 

  31. Sethi, T.S., Kantardzic, M.: On the reliable detection of concept drift from streaming unlabeled data. Expert Syst. Appl. 82, 77–99 (2017)

    Article  Google Scholar 

  32. Villalba-Diez, J., Schmidt, D., Gevers, R., Ordieres-Meré, J., Buchwitz, M., Wellbrock, W.: Deep learning for industrial computer vision quality control in the printing industry 4.0. Sensors 19(18), 3987 (2019)

    Google Scholar 

  33. Wankerl, H., Stern, M.L., Altieri-Weimar, P., Al-Baddai, S., Lang, K.J., Roider, F., Lang, E.W.: Fully convolutional networks for void segmentation in X-ray images of solder joints. J. Manuf. Process. 57, 762–767 (2020)

    Article  Google Scholar 

  34. Yang, J., Zhou, K., Li, Y., Liu, Z.: Generalized out-of-distribution detection: a survey. CoRR abs/2110.11334, https://arxiv.org/abs/2110.11334 (2021)

  35. Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., Chen, H.: Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=BJJLHbb0-

Download references

Author information

Authors and Affiliations

  1. University of Augsburg, Universitätsstraße 2, 86159, Augsburg, Germany

    Philipp Mascha

Authors
  1. Philipp Mascha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Mascha .

Editor information

Editors and Affiliations

  1. Computer Vision Laboratory, University of Sassari, Alghero, Sassari, Italy

    Massimo Tistarelli

  2. Computer Vision and Biometrics Lab, Department of Information Technology, Indian Institute of Information Technology Allahabad, Prayagraj, India

    Shiv Ram Dubey

  3. Computer Vision and Biometrics Lab, Department of Information Technology, Indian Institute of Information Technology, Allahabad, India

    Satish Kumar Singh

  4. University of Münster, Münster, Germany

    Xiaoyi Jiang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mascha, P. (2023). A Comparison of Model Confidence Metrics on Visual Manufacturing Quality Data. In: Tistarelli, M., Dubey, S.R., Singh, S.K., Jiang, X. (eds) Computer Vision and Machine Intelligence. Lecture Notes in Networks and Systems, vol 586. Springer, Singapore. https://doi.org/10.1007/978-981-19-7867-8_14

Download citation

Publish with us

Policies and ethics