Skip to main content
SpringerLink
Log in

Proportional data modeling via entropy-based variational bayes learning of mixture models

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

During the last few decades, many statistical approaches that were developed in the fields of computer vision and pattern recognition are based on mixture models. A mixture-based representation has a number of advantages: mixture models are generative, flexible, plus they can take prior information into account to improve the generalization capability. The mixture models that we consider in this paper are based on the Dirichlet and generalized Dirichlet distributions that have been widely used to represent proportional data. The novel aspect of this paper is to develop an entropy-based framework to learn these mixture models. Specifically, we propose a Bayesian framework for model learning by means of a sophisticated entropy-based variational Bayes technique. We present experimental results to show that the proposed method is effective in several applications namely person identity verification, 3D object recognition, text document clustering, and gene expression categorization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://conradsanderson.id.au/vidtimit

  2. http://www.cs.cmu.edu/~mccallum/bow/

  3. http://www.cs.cmu.edu/~textlearning/

  4. http://qwone.com/~jason/20Newsgroups/

References

  1. Alizadeh A, Eisen M, Davis R, et al (2000) Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling. Nature 403:503–511

    Article  Google Scholar 

  2. Ansary TF, Daoudi M, Vandeborre J (2007) A bayesian 3-d search engine using adaptive views clustering. IEEE Trans Multimedia 9(1):78–88

    Article  Google Scholar 

  3. Attias H (1999) A variational Bayes framework for graphical models. In: Proceedings of neural information processing systems (NIPS), pp 209–215

  4. Bahrampour S, Moshiri B, Salahshoor K (2011) Weighted and constrained possibilistic C-means clustering for online fault detection and isolation. Appl Intell 35(2):269–284

    Article  Google Scholar 

  5. Ben-David S, Kushilevitz E, Mansour Y (1997) Online learning versus offline learning. Mach Learn 29 (1):45–63

    Article  MATH  Google Scholar 

  6. Ben-Yacoub S, Abdeljaoued Y, Mayoraz E (1999) Fusion of face and speech data for person identity verification. IEEE Trans Neural Netw 10(5):1065–1074

    Article  Google Scholar 

  7. Bhanu B, Lin Y (2003) Genetic algorithm based feature selection for target detection in SAR images. Image Vision Comput 21(7):591–608

    Article  Google Scholar 

  8. Bimbo AD, Pala P (2006) Content-based retrieval of 3D models. ACM Trans Multimed Comput Commun Appl 2(1):20–43

    Article  Google Scholar 

  9. Bishop CM (2006) Pattern recognition and machine learning. Springer, Berlin

    MATH  Google Scholar 

  10. Bouguila N (2008) Clustering of count data using generalized Dirichlet multinomial distributions. IEEE Trans Knowl Data Eng 20(4):462–474

    Article  Google Scholar 

  11. Bouguila N, Ziou D (2006) A hybrid SEM algorithm for high-dimensional unsupervised learning using a finite generalized dirichlet mixture. IEEE Trans Image Process 15(9):2657–2668

    Article  Google Scholar 

  12. Bouguila N, Ziou D, Vaillancourt J (2004) Unsupervised learning of a finite mixture model based on the Dirichlet distribution and its application. IEEE Trans Image Process 13(11):1533–1543

    Article  Google Scholar 

  13. Boutemedjet S, Bouguila N, Ziou D (2009) A hybrid feature extraction selection approach for high-dimensional non-Gaussian data clustering. IEEE Trans Pattern Anal Mach Intell 31(8):1429–1443

    Article  Google Scholar 

  14. Bronstein M, Kokkinos I (2010) Scale-invariant heat kernel signatures for non-rigid shape recognition. In: Proceedings of the 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp 1704–1711

  15. Bustos B, Keim DA, Saupe D, Schreck T, Vranić DV (2005) Feature-based similarity search in 3D object databases. ACM Comput Surv 37(4):345–387

    Article  Google Scholar 

  16. Cardinaux F, Sanderson C, Bengio S (2006) User authentication via adapted statistical models of face images. IEEE Trans Signal Process 54(1):361–373

    Article  Google Scholar 

  17. Collins RT, Liu Y (2003) On-line selection of discriminative tracking features. In: 9th IEEE international conference on computer vision (ICCV 2003), 14–17 October 2003, Nice, France, pp 346–352

  18. Constantinopoulos C, Likas A (2007) Unsupervised learning of Gaussian mixtures based on variational component splitting. IEEE Trans Neural Netw 18(3):745–755

    Article  Google Scholar 

  19. Corduneanu A, Bishop CM (2001) Variational Bayesian model selection for mixture distributions. In: Proceedings of the 8th international conference on artificial intelligence and statistics (AISTAT), pp 27–34

  20. Daras P, Axenopoulos A (2010) A 3D shape retrieval framework supporting multimodal queries. Int J Comput Vis 89(2-3):229–247

    Article  Google Scholar 

  21. Faivishevsky L, Goldberger J (2009) ICA Based on a smooth estimation of the differential entropy. In: Koller D, Schuurmans D, Bengio Y, Bottou L (eds) Advances in neural information processing systems 21, pp 433–440

  22. Fan W, Bouguila N (2014) Online variational learning of generalized Dirichlet mixture models with feature selection. Neurocomputing 126:166–179

    Article  Google Scholar 

  23. Fan W, Bouguila N, Ziou D (2011) A variational statistical framework for object detection. In: 18th international conference on neural information processing, ICONIP 2011, pp 276–283

  24. Fan W, Bouguila N, Ziou D (2012) Variational learning for finite dirichlet mixture models and applications. IEEE Transactions on Neural Networks and Learning Systems 23(5):762–774

    Article  Google Scholar 

  25. Fan W, Bouguila N, Ziou D (2014) Variational learning of finite Dirichlet mixture models using component splitting. Neurocomputing 129:3–16

    Article  Google Scholar 

  26. Froba B, Ernst A (2004) Face detection with the modified census transform, pp 91–96

  27. Gao Y, Wang M, Zha Z, Tian Q, Dai Q, Zhang N (2011) Less is more: efficient 3-d object retrieval with query view selection. IEEE Trans Multimedia 13(5):1007–1018

    Article  Google Scholar 

  28. Gao Y, Wang M, Tao D, Ji R, Dai Q (2012) 3-D object retrieval and recognition with hypergraph analysis. IEEE Trans Image Process 21(9):4290–4303

    Article  MathSciNet  Google Scholar 

  29. Geusebroek JM, Burghouts GJ, Smeulders AWM (2005) The amsterdam library of object images. Int J Comput Vision 61(1):103–112

    Article  Google Scholar 

  30. McLachlan GJ, Peel D (2000) Finite mixture models. Wiley, New York

    Book  MATH  Google Scholar 

  31. Haering N, da Vitoria Lobo N (1999) Features and classification methods to locate deciduous trees in images. Comput Vis Image Underst 75(1):133–149

    Article  Google Scholar 

  32. Haigh J, Mason JS (1993) A voice activity detector based on cepstral analysis. In: Proceedings of the European conference on speech communication and technology, pp 1103–1106

  33. Hensman J, Rattray M, Lawrence ND (2015) Fast nonparametric clustering of structured time-series. IEEE Trans Pattern Anal Mach Intell 37(2):383–393

    Article  Google Scholar 

  34. Jain A, Ross A, Prabhakar S (2004) An introduction to biometric recognition. IEEE Trans Circuits Syst Video Technol 14(1):4–20

    Article  Google Scholar 

  35. Ji Y, Wu C, Liu P, Wang J, Coombes KR (2005) Applications of Beta-mixture models in bioinformatics. Bioinformatics 21(9):2118–2122

    Article  Google Scholar 

  36. Jordan MI, Ghahramani Z, Jaakkola TS, Saul LK (1999) An introduction to variational methods for graphical models. Mach Learn 37(2):183–233

    Article  MATH  Google Scholar 

  37. Kaya M, Alhajj R (2008) Online mining of fuzzy multidimensional weighted association rules. Appl Intell 29(1):13–34

    Article  Google Scholar 

  38. Kim H, Howland P, Park H (2005) Dimension reduction in text classification with support vector machines. J Mach Learn Res 6:37–53

    MathSciNet  MATH  Google Scholar 

  39. Kim S, Kweon I (2008) Scalable representation for 3D object recognition using feature sharing and view clustering. Pattern Recogn 41(2):754–773

    Article  MATH  Google Scholar 

  40. Kittler J, Hatef M, Duin R, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239

    Article  Google Scholar 

  41. Langley P, Provan GM, Smyth P (1997) Learning with probabilistic representations. Mach Learn 29 (2-3):91–101

    Article  MATH  Google Scholar 

  42. Leonenko N, Pronzato L, Savani V (2008) A class of rényi information estimators for multidimensional densities. Ann Stat 26(5):2153–2182

    Article  MATH  Google Scholar 

  43. Lewis D D, Yang Y, Rose T G, Li F (2004) RCV1: A new benchmark collection for text categorization research. J Mach Learn Res 5:361–397

    Google Scholar 

  44. Li X, Guskov I (2007) 3D object recognition from range images using pyramid matching. In: IEEE 11th international conference on computer vision, (ICCV 2007), pp 1–6

  45. Liang F, Liu C, Wang N (2007) A robust sequential bayesian method for identification of differentially expressed genes. Stat Sin 17:571–597

    MathSciNet  MATH  Google Scholar 

  46. Martino L, Read J, Luengo D (2015) Independent doubly adaptive rejection metropolis sampling within gibbs sampling. IEEE Trans Signal Process 63(12):3123–3138

    Article  MathSciNet  Google Scholar 

  47. Masoud H, Jalili S, Hasheminejad SMH (2013) Dynamic clustering using combinatorial particle swarm optimization. Appl Intell 38(3):289–314

    Article  Google Scholar 

  48. McLachlan GJ, Khan N (2004) On a resampling approach for tests on the number of clusters with mixture model-based clustering of tissue samples. J Multivar Anal 90(1):90–105

    Article  MathSciNet  MATH  Google Scholar 

  49. Nasios N, Bors AG (2006) Variational learning for Gaussian mixture models. IEEE Trans Syst Man Cybern B Cybern 36(4):849–862

    Article  Google Scholar 

  50. Nian R, Ji G, Zhao W, Feng C (2007) Probabilistic 3D object recognition from 2d invariant view sequence based on similarity. Neurocomputing 70(4-6):785–793

    Article  Google Scholar 

  51. Penalver A, Escolano F (2012) Entropy-based incremental variational Bayes learning of Gaussian mixtures. IEEE Transactions on Neural Networks and Learning Systems 23(3):534–540

    Article  Google Scholar 

  52. Poh N, Kittler J, Alkoot F (2012) A discriminative parametric approach to video-based score-level fusion for biometric authentication. In: Proceedings of the 21st international conference on pattern recognition (ICPR), pp 2335–2338

  53. Reynolds D (1994) Experimental evaluation of features for robust speaker identification. IEEE Transactions on Speech and Audio Processing 2(4):639–643

    Article  Google Scholar 

  54. Reynolds DA, Quatieri TF, Dunn RB (2000) Speaker verification using adapted gaussian mixture models. Digital Signal Process 10(1–3):19–41

    Article  Google Scholar 

  55. Robert C, Casella G (1999) Monte carlo statistical methods. Springer, Berlin

    Book  MATH  Google Scholar 

  56. Rotella F, Leuzzi F, Ferilli S (2015) Learning and exploiting concept networks with conNeKTion. Appl Intell 42(1):87–111

    Article  Google Scholar 

  57. Sanderson C, Lovell B (2009) Multi-region probabilistic histograms for robust and scalable identity inference. In: Tistarelli M, Nixon M (eds) Advances in biometrics, lecture notes in computer science, vol 5558, pp 199–208

  58. Sanderson C, Paliwal KK (2003) Fast features for face authentication under illumination direction changes. Pattern Recogn Lett 24(14):2409–2419

    Article  Google Scholar 

  59. Sanderson C, Paliwal KK (2004) Identity verification using speech and face information. Digital Signal Process 14(5):449–480

    Article  Google Scholar 

  60. Shih J, Lee C, Wang JT (2007) A new 3D model retrieval approach based on the elevation descriptor. Pattern Recogn 40(1):283–295

    Article  MATH  Google Scholar 

  61. Shilane P, Min P, Kazhdan M, Funkhouser T (2004) The princeton shape benchmark. In: Proceedings of the shape modeling international 2004, SMI ’04, pp 167–178

  62. Soong F, Rosenberg A (1988) On the use of instantaneous and transitional spectral information in speaker recognition. IEEE Trans Acoust Speech Signal Process 36(6):871–879

    Article  MATH  Google Scholar 

  63. Sun J, Ovsjanikov M, Guibas L (2009) A concise and provably informative multi-scale signature based on heat diffusion. In: Proceedings of the symposium on geometry processing, SGP ’09, pp 1383–1392

  64. Vien NA, Ertel W, Dang VH, Chung T (2013) Monte-carlo tree search for Bayesian reinforcement learning. Appl Intell 39(2):345–353

    Article  Google Scholar 

  65. Wildermoth BR, Paliwal KK (2000) Use of voicing and pitch information for speaker recognition. In: Proceedings of the 8th Australian international conference on speech science and technology, pp 324–328

  66. Yuan ZH, Lu T (2013) Incremental 3D reconstruction using Bayesian learning. Appl Intell 39(4):761–771

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This research was partially funded by the National Natural Science Foundation of China (61502183, 61673186), the Scientific Research Funds of Huaqiao University (600005-Z15Y0016), and the Natural Sciences and Engineering Research Council of Canada (NSERC). The second author is funded by a NSTIP KACST grant (13-INF1123-10),

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Huaqiao University, Xiamen, China

    Wentao Fan & Jixiang Du

  2. Department of Computer Engineering, College of Computer Systems, Umm Al-Qura University, Makkah, Saudi Arabia

    Faisal R. Al-Osaimi

  3. Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montreal, QC, Canada

    Nizar Bouguila

Authors
  1. Wentao Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Faisal R. Al-Osaimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nizar Bouguila
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jixiang Du
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wentao Fan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, W., Al-Osaimi, F.R., Bouguila, N. et al. Proportional data modeling via entropy-based variational bayes learning of mixture models. Appl Intell 47, 473–487 (2017). https://doi.org/10.1007/s10489-017-0909-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-017-0909-0

Keywords

Search

Navigation