Abstract
In this paper, we propose a novel normalization framework, multi-modal normalization(MultiNorm) that learns the multiple modalities through affine transformations involved in the normalization architecture. We have shown its effectiveness in speech-driven facial video generation and video emotion detection which are complex problems due to its multi-modal aspects in audio and video domain.
The multi-modal normalization uses various features such as mel spectrogram, pitch, energy from audio signals and predicted keypoint heatmap/ optical flow and a single image to learn the respective affine parameters to generate highly expressive video. The incorporation of multimodal normalization has given superior performances in the proposed video synthesis from an audio and a single image against previous methods on various metrics such as SSIM (structural similarity index), PSNR (peak signal to noise ratio), CPBD (image sharpness), WER (word error rate), blinks/sec and LMD (landmark distance). In video emotion detection, we have leveraged the mel-spectrogram and optical flow in the multi modal normalization to learn the affine parameters that helps in improving the accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Assael, Y.M., Shillingford, B., Whiteson, S., de Freitas, N.: Lipnet: end-to-end sentence-level lipreading. In: GPU Technology Conference (2017)
Ba, J., Kiros, J., Hinton, G.: Layer normalization (2016)
Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In: International Conference on Computer Vision (2017)
Cao, H., Cooper, D., Keutmann, M., Gur, R., Nenkova, A., Verma, R.: Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE Trans. affective Comput. 5, 377–390 (2014). https://doi.org/10.1109/TAFFC.2014.2336244
Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In: arXiv preprint arXiv:1812.08008 (2018)
Chen, L., Li, Z., Maddox, R.K., Duan, Z., Xu, C.: Lip movements generation at a glance (2018)
Chen, L., Maddox, R., Duan, Z., Xu, C.: Hierarchical cross-modal talking face generation with dynamic pixel-wise loss (2019)
Chung, J.S., Nagrani, A., Zisserman, A.: Voxceleb2: Deep speaker recognition. In: INTERSPEECH (2018)
Chung, J.S., Jamaludin, A., Zisserman, A.: You said that? In: British Machine Vision Conference (2017)
Cooke, M., Barker, J., Cunningham, S., Shao, X.: Grid AV speech corpus sample (2013)
Das, D., Biswas, S., Sinha, S., Bhowmick, B.: Speech-driven facial animation using cascaded GANs for learning of motion and texture. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12375, pp. 408–424. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58577-8_25
Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) SCIA 2003. LNCS, vol. 2749, pp. 363–370. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-45103-X_50
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, New York (2016). http://www.deeplearningbook.org
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp.1510–1519 (2017). https://doi.org/10.1109/ICCV.2017.167
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift (2015)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks (2019)
Kim, J., Kim, M., Kang, H.W., Lee, K.: U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation (2019)
Kingma, D., Ba, J.: Adam: A method for stochastic optimization. International Conference on Learning Representations (2014)
Kumar, N., Goel, S., Narang, A., Hasan, M.: Robust one shot audio to video generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2020)
Liu, D., Vu, M.T., Chatterjee, S., Rasmussen, L.: Neural network based explicit mixture models and expectation-maximization based learning. In: 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1–10 (2020)
Liu, G., Shih, K.J., Wang, T.C., Reda, F.A., Sapra, K., Yu, Z., Tao, A., Catanzaro, B.: Partial convolution based padding. arXiv preprint arXiv:1811.11718 (2018)
Mallya, A., Wang, T.C., Sapra, K., Liu, M.Y.: World-consistent video-to-video synthesis (2020)
Alghamdi, N., Maddock, S., Marxer, R., Barker, J., Brown, G.J..: A corpus of audio-visual lombard speech with frontal and profile view. The Journal of the Acoustical Society of America 143, EL523–EL529 (2018). https://doi.org/10.1121/1.5042758
Nam, H., Kim, H.E.: Batch-instance normalization for adaptively style-invariant neural networks (05 2018)
Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)
Pfülb, B., Gepperth, A.: Overcoming catastrophic forgetting with gaussian mixture replay. ArXiv abs/2104.09220 (2021)
Richardson, E., Weiss, Y.: On GANs and GMMs. In: NeurIPS (2018)
Soukupova, T., Cech, J.: Real-time eye blink detection using facial landmarks (2016)
Storey, G., Bouridane, A., Jiang, R., Li, C.-T.: Atypical facial landmark localisation with stacked hourglass networks: a study on 3D facial modelling for medical diagnosis. In: Jiang, R., Li, C.-T., Crookes, D., Meng, W., Rosenberger, C. (eds.) Deep Biometrics. USL, pp. 37–49. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-32583-1_3
Tulyakov, S., Liu, M.Y., Yang, X., Kautz, J.: MoCoGAN: Decomposing motion and content for video generation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1526–1535 (2018)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization (2016)
Vougioukas, K., Petridi, S., Pantic, M.: End-to-end speech-driven facial animation with temporal GANs. J. Foo 14(1), 234–778 (2004)
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: ECA-Net: Efficient channel attention for deep convolutional neural networks (2019)
Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
Wiles, O., Koepke, A., Zisserman, A.: X2face: A network for controlling face generation by using images, audio, and pose codes. In: European Conference on Computer Vision (2018)
Zhou, B., Khosla, A., A., L., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR (2016)
Zhu, H., Huang, H., Li, Y., Zheng, A., He, R.: Arbitrary talking face generation via attentional audio-visual coherence learning. arXiv: Computer Vision and Pattern Recognition (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kumar, N., Narang, A., lall, B., Goel, S. (2021). Multi Modal Normalization. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Lecture Notes in Computer Science(), vol 13108. Springer, Cham. https://doi.org/10.1007/978-3-030-92185-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-92185-9_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92184-2
Online ISBN: 978-3-030-92185-9
eBook Packages: Computer ScienceComputer Science (R0)