Abstract
We investigate the personalization of deep convolutional neural networks for facial expression analysis from still images. While prior work has focused on population-based (“one-size-fits-all”) approaches, we formulate and construct personalized models via a mixture of experts and supervised domain adaptation approach, showing that it improves greatly upon non-personalized models. Our experiments demonstrate the ability of the model personalization to quickly and effectively adapt to limited amounts of target data. We also provide a novel training methodology and architecture for creating personalized machine learning models for more effective analysis of emotion state.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For notational simplicity, we drop here the dependence on the source/target subjects.
- 2.
For instance, only the ResNet features of target subjects need be provided as input to the adapted model, as original face images cannot be reconstructed from those features.
- 3.
Note, however, that the Resnet used to extract the features for these models was fine-tuned using the labeled source data.
References
Peterson, K., Rudovic, O., Guerrero, R., Picard, R.W.: Personalized Gaussian processes for future prediction of Alzheimer’s disease progression. In: NIPS Workshop on Machine Learning for Healthcare (2017)
Jaques, N., Rudovic, O., Taylor, S., Sano, A., Picard, R.: Predicting tomorrow’s mood, health, and stress level using personalized multitask learning and domain adaptation. In: IJCAI Workshop (2017)
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE TPAMI 31, 39–58 (2009)
Rudovic, O., Lee, J., Dai, M., Schuller, B., Picard, R.: Personalized machine learning for robot perception of affect and engagement in autism therapy. arXiv preprint arXiv:1802.01186 (2018)
Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE TAC (2017)
Martinez, D.L., Rudovic, O., Picard, R.: Personalized automatic estimation of self-reported pain intensity from facial expressions. In: IEEE CVPR’W (2017)
Csurka, G.: Domain adaptation for visual applications: a comprehensive survey. CoRR, abs/1702.05374 (2017). http://arxiv.org/abs/1702.05374
Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural Comput. 3(1), 79–87 (1991)
Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In: IEEE FG (Workshops) (2013)
Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., Pantic, M.: Avec 2016: depression, mood, and emotion recognition workshop and challenge. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. ACM (2016)
Collobert, R., Bengio, S., Bengio, Y.: A parallel mixture of SVMS for very large scale problems. In: NIPS (2002)
Shahbaba, B., Neal, R.: Nonlinear models using Dirichlet process mixtures. JMLR 10, 1829–1850 (2009)
Theis, L., Bethge, M.: Generative image modeling using spatial LSTMS. In: NIPS (2015)
Yao, B., Walther, D., Beck, D., Fei-Fei, L.: Hierarchical mixture of classification experts uncovers interactions between brain regions. In: NIPS (2009)
Rasmussen, C.E., Ghahramani, Z.: Infinite mixtures of Gaussian process experts. In: NIPS (2002)
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In: ICLR (2017)
Jiang, H., Learned-Miller, E.: Face detection with the faster R-CNN. In: IEEE FG (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE CVPR (2016)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)
Jiang, J.: A literature survey on domain adaptation of statistical classifiers. 3 (2008). http://sifaka.cs.uiuc.edu/jiang4/domainadaptation/survey
Chollet, F., et al.: Keras (2015)
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: a system for large-scale machine learning. In: OSDI (2016)
Tzirakis, P., Zafeiriou, S., Schuller, B.W.: End2you-the imperial toolkit for multimodal profiling by end-to-end learning. arXiv preprint arXiv:1802.01115 (2018)
Acknowledgments
The work of O. Rudovic has been funded by the European Union H2020, Marie Curie Action - Individual Fellowship no. 701236 (EngageMe).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Feffer, M., Rudovic, O.(., Picard, R.W. (2018). A Mixture of Personalized Experts for Human Affect Estimation. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2018. Lecture Notes in Computer Science(), vol 10935. Springer, Cham. https://doi.org/10.1007/978-3-319-96133-0_24
Download citation
DOI: https://doi.org/10.1007/978-3-319-96133-0_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-96132-3
Online ISBN: 978-3-319-96133-0
eBook Packages: Computer ScienceComputer Science (R0)