Skip to main content
SpringerLink
Log in

A hybrid method for learning multi-dimensional Bayesian network classifiers based on an optimization model

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Bayesian networks, which have a solid mathematical basis as classifiers, take the prior information of samples into consideration. They have gained considerable popularity for solving classification problems. However, many real-world applications can be viewed as classification problems in which instances have to be assigned to a set of different classes at the same time. To address this problem, multi-dimensional Bayesian network classifiers (MBCs), which organize class and feature variables as three subgraphs, have recently been proposed. Because each subgraph has different structural restrictions, three different learning algorithms are needed. In this paper, we present for the first time an MBC learning algorithm based on an optimization model (MBC-OM) that is inspired by the constraint-based Bayesian network structure learning method. MBC-OM uses the chi-squared statistic and mutual information to estimate the dependence coefficients among variables, and these are used to construct an objective function as an overall measure of the dependence for a classifier structure. Therefore, the problem of searching for an optimal classifier becomes one of finding the maximum value of the objective function in feasible fields. We prove the existence and uniqueness of the numerical solution. Moreover, we validate our method on five benchmark data sets. Experimental results are competitive, and outperform state-of-the-art algorithms for multi-dimensional classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Cai Z, Sun S, Si S, Yannou B (2011) Identifying product failure rate based on a conditional Bayesian network classifier. Expert Syst Appl 38(5):5036–5043

    Article  Google Scholar 

  2. Hsieh NC, Hung LP (2011) A data driven ensemble classifier for credit scoring analysis. Expert Syst Appl 37(1):534–545

    Article  Google Scholar 

  3. Mukherjee S, Sharma N (2012) Intrusion Detection using Naive Bayes Classifier with Feature Reduction. Procedia Technol 4:119–128

    Article  Google Scholar 

  4. Friedman N, Geiger D, Goldszmidt M (1997) Bayesian network classifiers. Mach Learn 29(2−3):131–163

    Article  MATH  Google Scholar 

  5. Jiang L, Zhang H, Cai Z (2009) A Novel Bayes model: hidden Naive Bayes. IEEE Trans Knowl Data Eng 21(10):1361–1371

    Article  Google Scholar 

  6. Balamurugan AA, Rajaram R, Pramala S, Rajalakshmi S, Jeyendran, C, Dinesh Surya Prakash J (2011) NB+: An improved Naive Bayesian algorithm Knowledge-Based Systems, vol. 24, no. 5, pp 563-569

  7. Jiang L, Cai Z, Wang D, Zhang H (2012) Improving Tree augmented Naive Bayes for class probability estimation. Knowl-Based Syst 26:239–245

    Article  Google Scholar 

  8. Lpez-Cruz PL, Larranaga P, DeFelipe J, Bielza C (2014) Bayesian network modeling of the consensus between experts: An application to neuron classification. Int J Approx Reason 55:3–22

    Article  Google Scholar 

  9. Madden MG (2009) On the classification performance of TAN and general Bayesian networks, Knowledge-Based Systems, vol. 22,no. 7, pp. 489-495

  10. Zhang ML, Zhou ZH (2006) Multi-label neural networks with applications to functional genomics and text categorization. IEEE Trans Knowl Data Eng 18(10):1338–1351

    Article  Google Scholar 

  11. Ortigosa-Hernandez J, Rodriguez JD, Alzate L, Lucania M, Inza I, Lozano JA (2012) Approaching Sentiment Analysis by using semi-supervised learning of multi-dimensional classifiers. Neurocomputing 92:98–115

    Article  Google Scholar 

  12. Boutell MR, Luo J, Shen X, Brown CM (2004) Learning multi-label scene classification. Pattern Recogn 37(9):1757–1771

    Article  Google Scholar 

  13. Gopal S, Yang Y (2010) Multi-label classification with metalevel features, in Proc. 33rd SIGIR, Geneva Switzerland 315–322

  14. Sanden C, Zhang JZ (2011) Enhancing multi-label music genre classification through ensemble techniques, in Proc. 34th SIGIR, Beijing China 705–714

  15. Pang B, Lee L (2008) Opinion mining and sentiment analysis. Foundationsand Trends in Information Retrieva l2(1-2):1–135

    Article  Google Scholar 

  16. Rodriguez JD, Perez A, Lozano JA (2010) Sensitivity analysis of k-fold cross validation inprediction error estimation. IEEE Trans Pattern Anal Mach Intell 32(3):569–574

    Article  Google Scholar 

  17. Zhang ML, Zhou ZH (2014) A review on multi-label learning algorithms. IEEE Trans Knowl Data Eng 26(8):819–1837

    Article  Google Scholar 

  18. Zhang ML, Zhou ZH (2007) ML-KNN: a lazy learning approach to multi-label learning. Pattern Recognit 40(7):2038–2048

    Article  MATH  Google Scholar 

  19. van der Gaag LC, de Waal PR (2006) Multi-dimensional Bayesian network classifiers. In: Third European Conference on Probabilistic Graphical Models, pp. 107-114

  20. Sucar LE, Bielza C, Morales EF, Hernandez-Leal P (2014) Multi-label classification with Bayesian network-based chain Classifiers. Pattern Recogn Lett 41:14–22

    Article  Google Scholar 

  21. de Waal PR, van der Gaag LC (2007) Inference and learning in multi-dimensional Bayesian network classifiers. In: European Conference on Symbolic and Quantitative Approaches to Reasoning under Uncertainty Lecture Notes in Artificial Intelligence 4724 501–511

  22. Bielza C, Li G, Larranaga P (2011) Multi-dimensional classification with Bayesian networks. Int J Approx Reason 52:705–727

    Article  MATH  MathSciNet  Google Scholar 

  23. Rodriguez JD, Lozano JA (2008) Multi-objective learning of multi-dimensional Bayesian classifiers. In: Proceedings of the Eighth International Conference on Hybrid Intelligent Systems, pp. 501-506

  24. Borchani H, Bielza C, Martłnez-Martłn P, Larranaga P (2012) Markov blanket based approach for learning multi-dimensional Bayesian network classifiers: an application to predict the European quality of life-5dimensions (EQ-5D) from the 39-item Parkinson’s disease questionnaire (PDQ-39). J Biomed Inform 45:1175–1184

    Article  Google Scholar 

  25. Zaragoza JC, Sucar LE, Morales EF (2011) A two-step method to learn multidimensional Bayesian network classifiers based on mutual information measures. In: Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference (FLAIRS) AAAI Press, pp644–649

  26. Gasse M, Aussem A, Elghazel H (2014) A hybrid algorithm for Bayesian network structure learning with application to multi-label learning. Expert Syst Appl 41:6755–6772

    Article  Google Scholar 

  27. Borgelt C (2010) A conditional independence algorithm for learning undirected graphical models. J Comput Syst Sci 76(1):1–33

    Article  MathSciNet  Google Scholar 

  28. Zhang Y, Zhang W, Xie Y (2013) Improved heuristic equivalent search algorithm based on Maximal Information Coefficient for Bayesian Network Structure Learning. Neurocomputing 117:186–195

    Article  Google Scholar 

  29. de Campos LM (2006) A Scoring Function for Learning Bayesian Networks based on Mutual Information and Conditional Independence Tests. J Mach Learn Res 7(1):2149–2187

    MATH  MathSciNet  Google Scholar 

  30. Chen XW, Anantha G, Lin XT (2008) Improving Bayesian Network Structure Learning with Mutual Information-Based Node Ordering in the K2 Algorithm. IEEE Trans Knowl Data Eng 20(5):1–13

    Article  Google Scholar 

  31. Conover WJ (1999) Statistics, Practical Nonparametric John Wiley and Sons Inc.

  32. Rodriguez JD, Lozano JA (2010) Learning Bayesian Network Classifiers for Multi-dimensional Supervised Classification Problems by Means of a Multi-objective Approach, Technical Report EHU-KZAA-TR-3-2010. Department of Computer Science and Artificial Intelligence University of the Basque Country San Sebastian, Spain

    Google Scholar 

  33. Kullback S (1968) Information Theory and Statistics. Dover Publication

  34. Chartrand G, Zhang P (2005) Introduction to Graph Theory. McGraw-Hill College

  35. Neapolitan RE (2004) Learning Bayesian Networks. Prentice-Hall, Englewood Cliffs, NJ

    Google Scholar 

  36. Chickering DM, Meek C (2006) On the incompatibility of faithfulness and monotone DAG faithfulness. Artif Intell 170(8–9):653–666

    Article  MathSciNet  Google Scholar 

  37. http://bnj.sourceforge.net/

  38. http://mlkd.csd.auth.gr/multilabel.html.

Download references

Acknowledgments

The authors would like to thank the editor and the anonymous reviewers for their insightful comments and suggestions. This work has been partially supported by the National Natural Science Foundation of China (Grant No. 61373174, 11401454), the Natural Science Foundation of Shannxi Province, China (Grant No. 2014JQ1031) and the Fundamental Research Funds for the Central Universities (Grant No. JB140711, 72135992).

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xidian University, Xi’an, 710071, China

    Mingmin Zhu & Sanyang Liu

  2. School of Computer Science and Technology, Xidian University, Xi’an, 710071, China

    Jiewei Jiang

Authors
  1. Mingmin Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanyang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiewei Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingmin Zhu.

Appendix A

Appendix A

Table 3 Order-1 dependence coefficients for Emotions data set
Full size table
Table 4 Order-1 dependence coefficients for Emotions data set
Full size table
Table 5 Order-1 dependence coefficients for Emotions data set
Full size table
Table 6 Order-1 dependence coefficients for Emotions data set
Full size table
Table 7 Order-1 dependence coefficients for Emotions data set
Full size table
Table 8 Order-1 dependence coefficients for Emotions data set
Full size table
Table 9 Order-1 dependence coefficients for Emotions data set
Full size table
Table 10 Order-1 dependence coefficients for Emotions data set
Full size table
Table 11 Order-1 dependence coefficients for Emotions data set
Full size table
Table 12 Order-1 dependence coefficients for Emotions data set
Full size table
Table 13 Order-1 dependence coefficients for Emotions data set
Full size table
Table 14 Order-1 dependence coefficients for Emotions data set
Full size table
Table 15 Order-1 dependence coefficients for Emotions data set
Full size table
Table 16 Order-1 dependence coefficients for Emotions data set
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, M., Liu, S. & Jiang, J. A hybrid method for learning multi-dimensional Bayesian network classifiers based on an optimization model. Appl Intell 44, 123–148 (2016). https://doi.org/10.1007/s10489-015-0698-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-015-0698-2

Keywords

Search

Navigation