Abstract
In this study we address models with latent variable in the context of neural networks. We analyze a neural network architecture, mixture of deep experts (MoDE), that models latent variables using the mixture of expert paradigm. Learning the parameters of latent variable models is usually done by the expectation-maximization (EM) algorithm. However, it is well known that back-propagation gradient-based algorithms are the preferred strategy for training neural networks. We show that in the case of neural networks with latent variables, the back-propagation algorithm is actually a recursive variant of the EM that is more suitable for training neural networks. To demonstrate the viability of the proposed MoDE network it is applied to the task of speech presence probability estimation, widely applicable to many speech processing problem, e.g. speaker diarization and separation, speech enhancement and noise reduction. Experimental results show the benefits of the proposed architecture over standard fully-connected networks with the same number of parameters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The gradient of the incomplete-data likelihood can be calculated by the expectation of the complete-data likelihood by the Fisher identity.
References
Matlab software for speech enhancement based on optimally modified lsa (OMLSA) speech estimator and improved minima controlled recursive averaging (IMCRA) noise estimation approach for robust speech enhancement. http://webee.technion.ac.il/people/IsraelCohen/
Cappé, O., Moulines, E.: On-line expectation-maximization algorithm for latent data models. J. R. Stat. Soc. Ser. B (Stat. Method.) 71(3), 593–613 (2009)
Chazan, S.E., Gannot, S., Goldberger, J.: A phoneme-based pre-training approach for deep neural network with application to speech enhancement. In: 2016 IEEE International Workshop on Acoustic Signal Enhancement (IWAENC), pp. 1–5, September 2016
Cohen, I., Berdugo, B.: Noise estimation by minima controlled recursive averaging for robust speech enhancement. IEEE Sig. Process. Lett. 9(1), 12–15 (2002)
Cohen, I., Berdugo, B.: Speech enhancement for non-stationary noise environments. Sig. Process. 81(11), 2403–2418 (2001)
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. J. R. Stat. Soc. Ser. B (Methodol.) 39(1), 1–38 (1977). http://www.jstor.org/stable/2984875
Eigen, D., Ranzato, M., Sutskever, I.: Learning factored representations in a deep mixture of experts. In: International Conference on Learning Representations (ICLR), Workshop (2014)
Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural Comput. 3(1), 79–87 (1991)
Jordan, M.I., Jacobs, R.A.: Hierarchical mixtures of experts and the EM algorithm. Neural Comput. 6(2), 181–214 (1994)
Salakhutdinov, R., Roweis, S., Ghahramani, Z.: Optimization with EM and expectation-conjugate-gradient. In: International Conference on Machine Learning (ICML) (2003)
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In: International Conference on Learning Representations (ICLR) (2017)
Titterington, D.M.: Recursive parameter estimation using incomplete data. J. R. Stat. Soc. Ser. B 46, 257–267 (1984)
Varga, A., Steeneken, H.J.: Assessment for automatic speech recognition: NOISEX-92: a database and an experiment to study the effect of additive noise on speech recognition systems. Speech Commun. 12(3), 247–251 (1993)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Chazan, S.E., Gannot, S., Goldberger, J. (2018). Training Strategies for Deep Latent Models and Applications to Speech Presence Probability Estimation. In: Deville, Y., Gannot, S., Mason, R., Plumbley, M., Ward, D. (eds) Latent Variable Analysis and Signal Separation. LVA/ICA 2018. Lecture Notes in Computer Science(), vol 10891. Springer, Cham. https://doi.org/10.1007/978-3-319-93764-9_30
Download citation
DOI: https://doi.org/10.1007/978-3-319-93764-9_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-93763-2
Online ISBN: 978-3-319-93764-9
eBook Packages: Computer ScienceComputer Science (R0)