Skip to main content
SpringerLink
Log in

Loss Function with Memory for Trustworthiness Threshold Learning: Case of Face and Facial Expression Recognition

  • Conference paper
  • First Online:
Machine Learning, Optimization, and Data Science (LOD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13810))

Abstract

We compare accuracy metrics of the supervisor meta-learning artificial neural networks (ANN) that learn the trustworthiness of the Inception v.3 convolutional neural networks (CNN) ensemble prediction a priori of the “ground truth” verification on the face and facial expression recognition. One of the compared meta-learning ANN modes uses a simple majority of the ensemble votes and its predictions. In contrast, another uses dynamically learned “trusted” ensemble vote count and its a priori prediction to decide on the trustworthiness of the underlying CNN ensemble prediction. A custom loss function with memory is introduced to collect trustworthiness predictions and their errors during training. Based on the collected statistics, learning gradients for the “trusted” ensemble vote count parameter is calculated, and the “trusted” ensemble vote count threshold is dynamically determined. A facial data set with makeup and occlusion is used for computational experiments in the partition that ensures high out of the training data distribution conditions, where only non-makeup and non-occluded images are used for CNN model ensemble training, while the test set contains only makeup and occluded images.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Andrychowicz, M., et al.: Learning to learn by gradient descent by gradient descent. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS 2016, pp. 3988–3996. Curran Associates Inc., Red Hook (2016)

    Google Scholar 

  2. Baevski, A., Hsu, W.N., Xu, Q., Babu, A., Gu, J., Auli, M.: data2vec: a general framework for self-supervised learning in speech, vision and language (2022)

    Google Scholar 

  3. Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2021, pp. 610–623. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3442188.3445922

  4. Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 24. Curran Associates, Inc. (2011). https://proceedings.neurips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf

  5. Berretti, S., Del Bimbo, A., Pala, P., Amor, B.B., Daoudi, M.: A set of selected sift features for 3d facial expression recognition. In: 2010 20th International Conference on Pattern Recognition, pp. 4125–4128. IEEE (2010)

    Google Scholar 

  6. Blodgett, S.L., Madaio, M.: Risks of AI foundation models in education. CoRR abs/2110.10024 (2021). https://arxiv.org/abs/2110.10024

  7. Bommasani, R., et al.: On the opportunities and risks of foundation models. CoRR abs/2108.07258 (2021). https://arxiv.org/abs/2108.07258

  8. Brown, T.B., et al.: Language models are few-shot learners (2020)

    Google Scholar 

  9. Cacioppo, J.T., Berntson, G.G., Larsen, J.T., Poehlmann, K.M., Ito, T.A., et al.: The psychophysiology of emotion. Handbook of emotions 2(01), 2000 (2000)

    Google Scholar 

  10. Chen, C., Dantcheva, A., Swearingen, T., Ross, A.: Spoofing faces using makeup: an investigative study. In: 2017 IEEE International Conference on Identity, Security and Behavior Analysis, pp. 1–8 (Feb 2017)

    Google Scholar 

  11. Dietterich, T.G.: Ensemble methods in machine learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45014-9_1

    Chapter  Google Scholar 

  12. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)

    Article  Google Scholar 

  13. Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. CoRR abs/2101.03961 (2021). https://arxiv.org/abs/2101.03961

  14. Field, H.: At Stanford’s “foundation models” workshop, large language model debate resurfaces. Morning Brew, August 2021. https://www.morningbrew.com/emerging-tech/stories/2021/08/30/stanfords-foundation-models-workshop-large-language-model-debate-resurfaces

  15. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 1126–1135. PMLR, 06–11 Aug 2017. http://proceedings.mlr.press/v70/finn17a.html

  16. Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: representing model uncertainty in deep learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML2016, pp. 1050–1059. JMLR.org (2016)

    Google Scholar 

  17. Girdhar, R., Singh, M., Ravi, N., van der Maaten, L., Joulin, A., Misra, I.: Omnivore: A single model for many visual modalities (2022). 10.48550/ARXIV.2201.08377. https://arxiv.org/abs/2201.08377

  18. Graves, A.: Practical variational inference for neural networks. In: Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS 2011, pp. 2348–2356. Curran Associates Inc., Red Hook (2011)

    Google Scholar 

  19. Gross, C.T., Canteras, N.S.: The many paths to fear. Nat. Rev. Neurosci. 13(9), 651–658 (2012)

    Article  Google Scholar 

  20. Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. In: ‘Real-Life’ Images: Detection, Alignment, and Recognition. Erik Learned-Miller and Andras Ferencz and Frédéric Jurie, Marseille, France (2008)

    Google Scholar 

  21. Jakaite, L., Schetinin, V., Maple, C.: Bayesian assessment of newborn brain maturity from two-channel sleep electroencephalograms. Computational and Mathematical Methods in Medicine, pp. 1–7 (2012). https://doi.org/10.1155/2012/629654

  22. Jakaite, L., Schetinin, V., Maple, C., Schult, J.: Bayesian decision trees for EEG assessment of newborn brain maturity. In: The 10th Annual Workshop on Computational Intelligence UKCI 2010 (2010). https://doi.org/10.1109/UKCI.2010.5625584

  23. Jakaite, L., Schetinin, V., Schult, J.: Feature extraction from electroencephalograms for Bayesian assessment of newborn brain maturity. In: 24th International Symposium on Computer-Based Medical Systems (CBMS), pp. 1–6 (2011). https://doi.org/10.1109/CBMS.2011.5999109

  24. Jia, S., Li, X., Hu, C., Guo, G., Xu, Z.: 3d face anti-spoofing with factorized bilinear coding (2020)

    Google Scholar 

  25. Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? (2017). http://arxiv.org/abs/1703.04977

  26. Khodabakhsh, A., Busch, C.: A generalizable deepfake detector based on neural conditional distribution modelling. In: 2020 International Conference of the Biometrics Special Interest Group (BIOSIG), pp. 1–5 (2020)

    Google Scholar 

  27. Kim, B.-K., Roh, J., Dong, S.-Y., Lee, S.-Y.: Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J. Multimod. User Interfaces 10(2), 173–189 (2016). https://doi.org/10.1007/s12193-015-0209-0

    Article  Google Scholar 

  28. Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J.: Building machines that learn and think like people. Behav. Brain Sci. 40, e253 (2017). https://doi.org/10.1017/S0140525X16001837

    Article  Google Scholar 

  29. Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, pp. 6405–6416. Curran Associates Inc., Red Hook (2017)

    Google Scholar 

  30. Liu, M., Li, S., Shan, S., Chen, X.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126–136 (2015)

    Article  Google Scholar 

  31. Liu, X., Wang, X., Matwin, S.: Interpretable deep convolutional neural networks via meta-learning. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–9 (2018). https://doi.org/10.1109/IJCNN.2018.8489172

  32. Lopes, A.T., De Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2017)

    Article  Google Scholar 

  33. Lottick, K., Susai, S., Friedler, S.A., Wilson, J.P.: Energy usage reports: environmental awareness as part of algorithmic accountability. CoRR abs/1911.08354 (2019). http://arxiv.org/abs/1911.08354

  34. MacKay, D.J.C.: A practical bayesian framework for backpropagation networks. Neural Comput. 4(3), 448–472 (1992). https://doi.org/10.1162/neco.1992.4.3.448

    Article  Google Scholar 

  35. Mai, F., Pannatier, A., Fehr, F., Chen, H., Marelli, F., Fleuret, F., Henderson, J.: Hypermixer: An mlp-based green ai alternative to transformers. arXiv preprint arXiv:2203.03691 (2022)

  36. Mansourifar, H., Shi, W.: One-shot gan generated fake face detection (2020)

    Google Scholar 

  37. Marcus, G.: Deep learning: A critical appraisal. CoRR abs/1801.00631 (2018). http://arxiv.org/abs/1801.00631

  38. Martinez, A., Benavente, R.: The ar face database. Technical report 24, Computer Vision Center, Bellatera, June 1998

    Google Scholar 

  39. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE (2016)

    Google Scholar 

  40. Neal, R.M.: Bayesian learning for neural networks, Lecture Notes in Statistics, vol. 118. Springer-Verlag New York, Inc. (1996). https://doi.org/10.1007/978-1-4612-0745-0

  41. Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms. ArXiv abs/1803.02999 (2018)

    Google Scholar 

  42. Ram, R., Müller, S., Pfreundt, F., Gauger, N., Keuper, J.: Scalable hyperparameter optimization with lazy gaussian processes. 2019 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC), pp. 56–65 (2019)

    Google Scholar 

  43. Rosset, C.: Turing-NLG: A 17-billion-parameter language model by Microsoft - Microsoft Research, February 2020. https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft. Accessed 16 Jan 2022

  44. Schetinin, V., Jakaite, L., Krzanowski, W.: Bayesian averaging over decision tree models: an application for estimating uncertainty in trauma severity scoring. Int. J. Med. Informatics 112, 6–14 (2018). https://doi.org/10.1016/j.ijmedinf.2018.01.009

    Article  Google Scholar 

  45. Schetinin, V., Jakaite, L., Krzanowski, W.: Bayesian averaging over decision tree models for trauma severity scoring. Artif. Intell. Med. 84, 139–145 (2018). https://doi.org/10.1016/j.artmed.2017.12.003

    Article  Google Scholar 

  46. Schetinin, V., Jakaite, L., Krzanowski, W.: Bayesian learning of models for estimating uncertainty in alert systems: application to air traffic conflict avoidance. Integrated Comput.-Aided Eng. 26, 1–17 (2018). https://doi.org/10.3233/ICA-180567

    Article  Google Scholar 

  47. Schick, T., Schütze, H.: It’s not just size that matters: small language models are also few-shot learners. CoRR abs/2009.07118 (2020). https://arxiv.org/abs/2009.07118

  48. Selitskaya, N., Sielicki, S., Christou, N.: Challenges in real-life face recognition with heavy makeup and occlusions using deep learning algorithms. In: Nicosia, G., Ojha, V., La Malfa, E., Jansen, G., Sciacca, V., Pardalos, P., Giuffrida, G., Umeton, R. (eds.) LOD 2020. LNCS, vol. 12566, pp. 600–611. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64580-9_49

    Chapter  Google Scholar 

  49. Selitskaya, N., Sielicki, S., Christou, N.: Challenges in face recognition using machine learning algorithms: case of makeup and occlusions. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) IntelliSys 2020. AISC, vol. 1251, pp. 86–102. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-55187-2_9

    Chapter  Google Scholar 

  50. Selitskiy, S., Christou, N., Selitskaya, N.: Isolating Uncertainty of the Face Expression Recognition with the Meta-Learning Supervisor Neural Network, pp. 104–112. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3480433.3480447

  51. Selitskiy, S., Christou, N., Selitskaya, N.: Using statistical and artificial neural networks meta-learning approaches for uncertainty isolation in face recognition by the established convolutional models. In: Nicosia, G., Ojha, V., La Malfa, E., La Malfa, G., Jansen, G., Pardalos, P.M., Giuffrida, G., Umeton, R. (eds.) Machine Learning, Optimization, and Data Science, pp. 338–352. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-030-95470-3_26

    Chapter  Google Scholar 

  52. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  53. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., Kiela, D.: Flava: A foundational language and vision alignment model (2021). https://doi.org/10.48550/ARXIV.2112.04482. https://arxiv.org/abs/2112.04482

  54. Sprent, P.: Applied Nonparametric Statistical Methods. Springer, Dordrecht (1989)

    MATH  Google Scholar 

  55. Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. CoRR abs/1906.02243 (2019). http://arxiv.org/abs/1906.02243

  56. Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)

  57. Thrun S., P.L.: Learning To Learn. Springer, Boston, MA (1998). https://doi.org/10.1007/978-1-4615-5529-2

  58. Vanschoren, J.: Meta-learning: a survey. ArXiv abs/1810.03548 (2018)

    Google Scholar 

  59. Whitehill, J., Omlin, C.W.: Haar features for facs au recognition. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pp. 5-pp. IEEE (2006)

    Google Scholar 

  60. Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., Xia, W.: Learning to recognize patch-wise consistency for deepfake detection (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, University of Bedfordshire, Park Square, Luton, LU1 3JU, UK

    Stanislav Selitskiy

  2. IEEE Member, Georgia, USA

    Natalya Selitskaya

Authors
  1. Stanislav Selitskiy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Natalya Selitskaya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stanislav Selitskiy .

Editor information

Editors and Affiliations

  1. University of Catania, Catania, Italy

    Giuseppe Nicosia

  2. University of Reading, Reading, UK

    Varun Ojha

  3. University of Oxford, Oxford, UK

    Emanuele La Malfa

  4. University of Cambridge, Cambridge, UK

    Gabriele La Malfa

  5. University of Florida, Gainesville, FL, USA

    Panos Pardalos

  6. Free University of Bozen-Bolzano, Bolzano, Italy

    Giuseppe Di Fatta

  7. University of Catania, Catania, Italy

    Giovanni Giuffrida

  8. Dana-Farber Cancer Institute, Boston, MA, USA

    Renato Umeton

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Selitskiy, S., Selitskaya, N. (2023). Loss Function with Memory for Trustworthiness Threshold Learning: Case of Face and Facial Expression Recognition. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2022. Lecture Notes in Computer Science, vol 13810. Springer, Cham. https://doi.org/10.1007/978-3-031-25599-1_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25599-1_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25598-4

  • Online ISBN: 978-3-031-25599-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation