Abstract
Multiple studies in the past have shown that there is a strong correlation between human vocal characteristics and facial features. However, existing approaches generate faces simply from voice, without exploring the set of features that contribute to these observed correlations. A computational methodology to explore this can be devised by rephrasing the question to: “how much would a target face have to change in order to be perceived as the originator of a source voice?” With this in perspective, we propose a framework to morph a target face in response to a given voice in a way that facial features are implicitly guided by learned voice-face correlation in this paper. Our framework includes a guided autoencoder that converts one face to another, controlled by a unique model-conditioning component called a gating controller which modifies the reconstructed face based on input voice recordings. We evaluate the framework on VoxCelab and VGGFace datasets through human subjects and face retrieval. Various experiments demonstrate the effectiveness of our proposed model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Zhu, L.L., Beauchamp, M.: Mouth and voice: a relationship between visual and auditory preference in the human superior temporal sulcus. J. Neurosci. Official J. Soc. Neurosci. 37, 02 (2017)
Kim, C., Shin, H.V., Oh, T.-H., Kaspar, A., Elgharib, M., Matusik, W.: On learning associations of faces and voices. In: Proceedings of Asian Conference on Computer Vision (ACCV) (2018)
Oh, T.-H., et al.: Speech2face: learning the face behind a voice. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
Sadoughi, N., Busso, C.: Speech-driven expressive talking lips with conditional sequential generative adversarial networks. CoRR, vol. abs/1806.00154 (2018)
Wiles, C., Koepke, A., Zisserman, A.: X2face: a network for controlling face generation by using images, audio, and pose codes. In: European Conference on Computer Vision (2018)
Wen, Y., Ismail, M.A., Liu, W., Raj, B., Singh, R.: Disjoint mapping network for cross-modal matching of voices and faces. In: International Conference on Learning Representations (2019)
Nagrani, A., Albanie, S., Zisserman, A.: Seeing voices and hearing faces: cross-modal biometric matching. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
Duarte, A., et al.: “Wav2pix: speech-conditioned face generation using generative adversarial networks. In: 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (2019)
Li, H., Li, B., Tan, S., Huang, J.: Detection of deep network generated images using disparities in color components. arXiv e-prints, p. arXiv:1808.07276, August 2018
Wen, Y., Raj, B., Singh, R.: Face reconstruction from voice using generative adversarial networks. In: Advances in Neural Information Processing Systems, pp. 5266–5275 (2019)
Bian, Y., Wang, J., Jun, J.J., Xie, X.-Q.: Deep convolutional generative adversarial network (dcgan) models for screening and design of small molecules targeting cannabinoid receptors. Molecular pharmaceutics 16(11), 4451–4460 (2019)
Chen, H., Lu, C.: Nested variance estimating vae/gan for face generation. In: International Joint Conference on Neural Networks (IJCNN) 2019, pp. 1–8 (2019)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., Choo, J.: Stargan: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
Zhu, J., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision (ICCV) 2017, pp. 2242–2251 (2017)
Nagrani, A., Chung, J.S., Zisserman, A.: Voxceleb: a large-scale speaker identification dataset. In: INTERSPEECH (2017)
Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: bmvc, vol. 1, no. 3, p. 6 (2015)
Borji, A.: Pros and cons of gan evaluation measures. Comput. Vision Image Understanding 179, 41–65 (2019)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems 30, pp. 6626–6637 (2017)
Zhou, S., Gordon, M.L., Krishna, R., Narcomey, A., Li, F., Bernstein, M.: HYPE: a benchmark for human eye perceptual evaluation of generative models. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 December 2019, Vancouver, BC, Canada, pp. 3444–3456 (2019)
Bau, D., et al.: Gan dissection: visualizing and understanding generative adversarial networks. In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)
Lin, K., Li, D., He, X., Zhang, Z., Sun, M.-T.: Adversarial ranking for language generation, ser. NIPS’17, pp. 3158–3168 (2017)
Yu, L., Zhang, W., Wang, J., Yu, Y.: Seqgan: sequence generative adversarial nets with policy gradient. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, ser. AAAI 2017. AAAI Press, pp. 2852–2858 (2017)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Liang, H., Yu, L., Xu, G., Raj, B., Singh, R. (2020). Controlled AutoEncoders to Generate Faces from Voices. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science(), vol 12509. Springer, Cham. https://doi.org/10.1007/978-3-030-64556-4_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-64556-4_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64555-7
Online ISBN: 978-3-030-64556-4
eBook Packages: Computer ScienceComputer Science (R0)