Abstract
Finite Mixture Models are flexible models with varying uses such as density estimation, clustering, classification, modeling heterogeneity, model averaging, and handling missing data. One of the prerequisites of using mixture models is the a priori knowledge of the number of mixture components so that the Expectation Maximization (EM) algorithm can learn the maximum likelihood parameters of the mixture model. However, the number of mixing components is often unknown and determining the number of mixture components has been a central problem in mixture modelling. Thus, mixture modelling is often a two-stage process of determining the number of mixture components and then estimating the parameters of the mixture model. This paper proposes a fast, search-based model selection algorithm for mixture models using progressive merging of mixture components. The paper also proposes a data driven, fast approximation of the Kullback-Leibler (KL) divergence as a criterion to merge the mixture components. The proposed methodology is used in mixture modelling of two chromosomal aberration datasets showing that model selection is efficient and effective.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
McLachlan, G.J., Peel, D.: Finite mixture models. Wiley, New York (2000)
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal Of The Royal Statistical Society, Series B 39(1), 1–38 (1977)
Tikka, J., Hollmén, J., Myllykangas, S.: Mixture Modeling of DNA Copy Number Amplification Patterns in Cancer. In: Sandoval, F., Prieto, A.G., Cabestany, J., Graña, M. (eds.) IWANN 2007. LNCS, vol. 4507, pp. 972–979. Springer, Heidelberg (2007)
Smyth, P.: Model selection for probabilistic clustering using cross-validated likelihood. Statistics and Computing 10, 63–72 (2000)
Figueiredo, M.A.T., Jain, A.K.: Unsupervised Learning of Finite Mixture Models. IEEE Transactions on Pattern Anal. Mach. Intell. 24(3), 381–396 (2002)
Hollmén, J., Tikka, J.: Compact and Understandable Descriptions of Mixtures of Bernoulli Distributions. In: Berthold, M.R., Shawe-Taylor, J., Lavrač, N. (eds.) IDA 2007. LNCS, vol. 4723, pp. 1–12. Springer, Heidelberg (2007)
Adhikari, P.R., Hollmén, J.: Patterns from multi-resolution 0-1 data. In: Goethals, B., Tatti, N., Vreeken, J. (eds.) Proceedings of the ACM SIGKDD Workshop on Useful Patterns (UP 2010), pp. 8–12. ACM (July 2010)
Adhikari, P.R., Hollmén, J.: Preservation of Statistically Significant Patterns in Multiresolution 0-1 Data. In: Dijkstra, T.M.H., Tsivtsivadze, E., Marchiori, E., Heskes, T. (eds.) PRIB 2010. LNCS, vol. 6282, pp. 86–97. Springer, Heidelberg (2010)
Ueda, N., Nakano, R., Ghahramani, Z., Hinton, G.E.: SMEM Algorithm for Mixture Models. Neural Computation 12(9), 2109–2128 (2000)
Zhang, Z., Chen, C., Sun, J., Chan, K.L.: EM algorithms for Gaussian mixtures with split-and-merge operation. Pattern Recognition 36(9), 1973–1983 (2003)
Li, Y., Li, L.: A Novel Split and Merge EM Algorithm for gaussian mixture model. In: Fifth International Conference on Natural Computation, ICNC 2009, vol. 6, pp. 479–483 (August 2009)
Zhang, B., Zhang, C., Yi, X.: Competitive EM algorithm for finite mixture models. Pattern Recognition 37(1), 131–144 (2004)
Blekas, K., Lagaris, I.E.: Split–Merge Incremental LEarning (SMILE) of Mixture Models. In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D.P. (eds.) ICANN 2007. LNCS, vol. 4669, pp. 291–300. Springer, Heidelberg (2007)
Akaike, H.: A new look at the statistical model identification. IEEE Transactions on Automatic Control 19(6), 716–723 (1974)
Windham, M.P., Cutler, A.: Information Ratios for Validating Mixture Analyses. Journal of the American Statistical Association 87(420), 1188–1192 (1992)
Juang, B.H., Rabiner, L.R.: A probabilistic distance measure for Hidden Markov models. AT&T Technical Journal 64(2), 391–408 (1985)
Li, Y., Li, L.: A split and merge EM algorithm for color image segmentation. In: IEEE ICIS 2009, vol. 4, pp. 395–399 (November 2009)
Goldberger, J., Gordon, S., Greenspan, H.: An Efficient Image Similarity Measure Based on Approximations of KL-Divergence Between Two Gaussian Mixtures. In: Proceedings of the ICCV 2003, Washington DC, USA, pp. 487–493 (2003)
Hershey, J.R., Olsen, P.A.: Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models. In: IEEE. ICASSP 2007, vol. 4, pp. 317–320 (2007)
Mclachlan, G.J., Krishnan, T.: The EM Algorithm and Extensions, 1st edn., November 1996. Wiley Interscience (November 1996)
Kullback, S., Leibler, R.A.: On Information and Sufficiency. Annals of Mathematical Statistics 22(1), 79–86 (1951)
Jeffreys, H.: An Invariant Form for the Prior Probability in Estimation Problems. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences 186(1007), 453–461 (1946)
Beeferman, D., Berger, A.: Agglomerative clustering of a search engine query log. In: Proceedings of the ACM KDD 2000, New York, USA, pp. 407–416 (2000)
Myllykangas, S., Tikka, J., Böhling, T., Knuutila, S., Hollmén, J.: Classification of human cancers based on DNA copy number amplification modeling. BMC Medical Genomics 1(15) (May 2008)
Baudis, M.: Genomic imbalances in 5918 malignant epithelial tumors: an explorative meta-analysis of chromosomal CGH data. BMC Cancer 7, 226 (2007)
Kittler, J.: Feature selection and extraction. Handbook of Pattern Recognition and Image Processing. Academic Press (1986)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Adhikari, P.R., Hollmén, J. (2012). Fast Progressive Training of Mixture Models for Model Selection. In: Ganascia, JG., Lenca, P., Petit, JM. (eds) Discovery Science. DS 2012. Lecture Notes in Computer Science(), vol 7569. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33492-4_17
Download citation
DOI: https://doi.org/10.1007/978-3-642-33492-4_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33491-7
Online ISBN: 978-3-642-33492-4
eBook Packages: Computer ScienceComputer Science (R0)