Skip to main content
SpringerLink
Log in

ABC-Miner+: constructing Markov blanket classifiers with ant colony algorithms

  • Regular research paper
  • Published:
Memetic Computing Aims and scope Submit manuscript

Abstract

ABC-Miner is a Bayesian classification algorithm based on the Ant colony optimization (ACO) meta-heuristic. The algorithm learns Bayesian network Augmented Naïve-Bayes (BAN) classifiers, where the class node is the parent of all the nodes representing the input variables. However, this assumes the existence of a dependency relationship between the class variable and all the input variables, and this relationship is always a type of “causal” (rather than “effect”) relationship, which restricts the flexibility of the algorithm to learn. In this paper, we extended the ABC-Miner algorithm to be able to learn the Markov blanket of the class variable. Such a produced model has a more flexible Bayesian network classifier structure, where it is not necessary to have a (direct) dependency relationship between the class variable and each of the input variables, and the dependency between the class and the input variables varies from “causal” to “effect” relationships. In this context, we propose two algorithms: \({\hbox {ABC-Miner}+_1}\), in which the dependency relationships between the class and the input variables are defined in a separate phase before the dependency relationships among the input variables are defined, and \({\hbox {ABC-Miner}+_2}\), in which the two types of dependency relationships in the Markov blanket classifier are discovered in a single integrated process. Empirical evaluations on 33 UCI benchmark datasets show that our extended algorithms outperform the original version in terms of predictive accuracy, model size and computational time. Moreover, they have shown a very competitive performance against other well-known classification algorithms in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Asuncion A, Newman DJ (2007) UCI machine learning repository. URL:http://www.ics.uci.edu/mlearn/MLRepository.html. Accessed Jan 2012

  2. Buntine W (1991) Theory refinement on Bayesian networks. In 17th Conference on uncertainty in artificial intelligence, San Francisco, CA, USA, Morgan Kaufmann pp 52–60

  3. Cheng J, Greiner R (1999) Comparing Bayesian network classifiers. In 15th Annual conference on uncertainty in artificial intelligence, San Francisco, CA, USA, Morgan Kaufmann pp 101–108

  4. Cheng J, Greiner R (2001) Learning Bayesian belief network classifiers: algorithms and system. In 14th Biennial Conference of the Canadian Society on computational studies of intelligence: advances in artificial intelligence, Springer, London, UK, pp 141–151

  5. Chickering D, Geiger M, Heckerman D (1994) Learning Bayesian networks is NP-Hard. Microsoft Corporation, Technical Report, Advanced Technologies Division

  6. Chickering D, Heckerman D, Meek C (2004) Large-sample learning of Bayesian networks is NP-hard. J Mach Learn Res 5:1287–1330

    MATH  MathSciNet  Google Scholar 

  7. Cooper GF, Herskovits E (1992) A Bayesian method for the induction of probabilistic networks from data. Mach Learn 9(4):309–347

    MATH  Google Scholar 

  8. Daly R, Shen Q (2009) Learning Bayesian network equivalence classes with ant colony optimization. J Artif Intell Res (JAIR) 35:391–447

    MATH  MathSciNet  Google Scholar 

  9. Daly R, Shen Q, Aitken S (2011) Review: learning Bayesian networks: approaches and issues. Knowl Eng Rev 26(2):99–157

    Article  Google Scholar 

  10. de Campos LM, Fernandez-Luna JM, Gamez JA, Puerta JM (2002) Ant colony optimization for learning Bayesian networks. Int J Approx Reason 31(3):291–311

    Article  MATH  Google Scholar 

  11. Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 1(7):1–30

    MathSciNet  Google Scholar 

  12. Dorigo M, Caro GM, Gambardella LM (1999) Ant Algorithms Discrete Optim. Artif Life 5(2):137–172

    Article  Google Scholar 

  13. Dorigo M, Stützle T (2003) OPRMS., The ant colony optimization metaheuristic: algorithms, applications, and advancesSpringer, New York

    Google Scholar 

  14. Dorigo M, Stützle T (2004) Ant colony optimization. MIT Press, Cambridge

    Book  MATH  Google Scholar 

  15. Richard OD, Peter EH (1973) Pattern classification and scene analysis. Wiley, New York

    MATH  Google Scholar 

  16. Freitas AA (2013) Comprehensible classification models: a position paper. ACM SIGKDD Explor 15(1):1–10

    Article  MathSciNet  Google Scholar 

  17. Freitas AA, Wieser DC (2010) Apweiler R On the importance of comprehensible classification models for protein function prediction. IEEE/ACM Trans Comput Biol Bioinform 7(1):172–182

    Article  Google Scholar 

  18. Friedman N, Geiger D, Goldszmidt M, Provan G, Langley P, Smyth P (1997) Bayesian network classifiers. Mach Learn 29:131–163

    Article  MATH  Google Scholar 

  19. Friedman N, Goldszmidt M (1998) Learning Bayesian networks with local structure. Learning in graphical models. Kluwer, Norwell, pp 252–262

    Google Scholar 

  20. Garca S, Herrera F (2008) An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons. J Mach Learn Res 9:2677–2694

    Google Scholar 

  21. Gurwicz Y, Lerner B (2006) Bayesian class-matched multinet classifier. In International Conference on structural, syntactic, and statistical pattern recognition (IAPR’6), Berlin, Heidelberg, pp 145–153

  22. Han J, Kamber M (2000) Data mining: concepts and techniques, 2nd edn. Morgan Kaufmann, San Francisco

    Google Scholar 

  23. Heckerman D (2008) A Tutorial on learning with Bayesian networks. Studies in computational intelligence: innovations in Bayesian networks, vol 156. Springer, Berlin, Heidelberg, pp 33–82

  24. Heckerman David, Geiger Dan, Chickering David M (1995) Learning Bayesian networks: the combination of knowledge and statistical data. Mach Learn 20(3):197–243

    MATH  Google Scholar 

  25. Huang K, King I, Lyu MR (2003) Discriminative training of Bayesian Chow-Liu multinet classifiers. In International Joint Conference on Networks, vol 1, New York, NY, USA, IEEE Press, pp 484–488

  26. Japkowicz N, Shah M (2011) Evaluating learning algorithms: a classification perspective. Cambridge University Press, Cambridge

    Book  Google Scholar 

  27. Jiang L, Zhang H, Cai Z, Su J (2005) Evolutional Naive Bayes. In 1st International Symposium on Intelligent computation and its applications. China University of Geosciences Press, pp 344–350

  28. Jiang L,Wang D , Cai Z ,Yan X (2007) Survey of improving Naive Bayes for classification. In 3rd International Conference on Advanced data mining and applications (ADMA’07), number 4632 in LNCS, Berlin, Heidelberg, Springer, pp 134–145

  29. Silla CN Jr, Freitas AA (2011) A survey of hierarchical classification across different application domains. Data Min Knowl Discov 22(1–2):31–72

    Article  MATH  MathSciNet  Google Scholar 

  30. Santos E, Jr., Hussein A (2004) Case-based Bayesian network classifiers. In 17th International FLAIRS Conference, AAAI, vol 5, Stanford, USA, pp 598–605

  31. Kittler J (1986) Handbook of pattern recognition and image processing. Academic Press, New York

    Google Scholar 

  32. Korb KB, Nicholson AE (2011) Bayesian artificial intelligence, 2nd edn. CRC Press, San Francisco

    MATH  Google Scholar 

  33. Langley P (1993) Induction of recursive Bayesian classifiers. In European Conference on Machine learning (ECML), Berlin, Heidelberg, pp 153–164

  34. Langley P, Sage S (1994) Induction of selective Bayesian classifiers. In 10th Conference on Uncertainty in artificial intelligence, San Francisco, CA, USA, Kaufmann, pp 399–406

  35. Liu H, Motoda H (1998) Feature extraction, construction and selection: a data mining perspective, 1st edn. Springer, Berlin, Heidelber

    Book  MATH  Google Scholar 

  36. Marteens D, Vanthienen J, Verbeke W, Baesens B (2011) Performance of classification models from a user perspective. Decis Support Syst 51(4):782–793

    Article  Google Scholar 

  37. Martens D, De Backer M, Haesen R, Vanthienen J, Snoeck M, Baesens B (2007) Classification with ant colony optimization. IEEE Trans Evol Comput 11:651–665

    Article  Google Scholar 

  38. Mitchell TM (1980) The need for biases in learning generalizations. Read Mach Learn 10:184–191

    Google Scholar 

  39. Otero FE, Freitas AA, Johnson CG (2009) Handling continuous attributes in ant colony classification algorithms. In IEEE Symposium on Computational intelligence in data mining (CIDM 2009), New York, NY, USA, IEEE Press, pp 225–231

  40. Parpinelli RS, Lopes HS, Freitas AA (2002) Data mining with an ant colony optimization algorithm. IEEE Trans Evol Comput 6(4):321–332

    Article  Google Scholar 

  41. Pearl J (2000) Causality: models. Reasoning and inference. Cambridge University Press, Cambridge

    Google Scholar 

  42. Pinto Pedro C, Andreas Nägele, Dejori Mathäus, Runkler Thomas A, Ao (2009) Using a local discovery ant algorithm for Bayesian network structure learning. IEEE Trans Evol Comput 13(4):767–779

    Article  Google Scholar 

  43. Salama KM, Abdelbar AM, Freitas AA (2011) Multiple pheromone types and other extensions to the ant-miner classification rule discovery algorithm. Swarm Intell 5(3–4):149–182

    Article  Google Scholar 

  44. Salama KM, Abdelbar AM, Otero FE, Freitas AA (2013) Utilizing multiple pheromones in an ant-based algorithm for continuous-attribute classification rule discovery. Appl Soft Comput 13(1):667–675

    Article  Google Scholar 

  45. Salama KM, Freitas AA (2012) ABC-Miner: an ant-based Bayesian classification algorithm. In 8th International Conference on Swarm Intelligence (ANTS’12), number 7461 in LNCS, Berlin, Springer, pp 13–24

  46. Salama KM, Freitas AA (2013) ACO-based Bayesian Network ensembles for the hierarchical classification of ageing-related proteins. In The European Conference on Evolutionary computation, machine learning and data mining in computational biology (EvoBio’13), number 7833 in LNCS, Berlin, Heidelberg, Springer, pp 80–91

  47. Salama KM, Freitas AA (2013) Clustering-based Bayesian multi-net classifier construction with ant colony optimization. In IEEE Congress on Evolutionary computation (IEEE CEC) (2013), New York, NY, USA, pp 3079–3086

  48. Salama KM, Freitas AA (2013) Extending the ABC-Miner Bayesian classification algorithm. In 6th International Workshop on Nature inspired cooperative strategies for optimization (NICSO’13), vol 512 of Studies in Computational Intelligence, Berlin, 2013. Springer, pp 1–12

  49. Salama KM, Freitas AA (2013) Learning Bayesian network classifiers using ant colony optimization. Swarm Intell 7(2–3):229–254

    Article  Google Scholar 

  50. Santos E, Hussein A (2004) Comparing case-based bayesian network and recursive bayesian multi-net classifiers. In International Conference on Artificial intelligence (ICAI), pp 627–633

  51. Smaldon J, Freitas AA (2006) A new version of the ant-miner algorithm discovering unordered rule sets. In Genetic and Evolutionary Computation Conference (GECCO’06). ACM Press, pp 43–50

  52. Tan P-N, Steinbach M, Kumar V (2005) Introduction to data mining, 2nd edn. Addison Wesley, USA

    Google Scholar 

  53. Witten IH, Frank E (2010) Data mining: practical machine learning tools and techniques, 3rd edn. Morgan Kaufmann, San Francisco

    Google Scholar 

  54. Wu Y, McCall J, Corne D (2010) Two novel ant colony optimization approaches for Bayesian network structure learning. In IEEE Congress on Evolutionary computation (CEC), New York, NY, USA, . IEEE Press, pp 1–7

  55. Yang S, Chang K-C (2002) Comparison of score metrics for Bayesian network learning. IEEE Trans Syst Man Cyber Part A 32(3):419–428

    Article  Google Scholar 

  56. Zheng F, Webb GI (2008) Semi-naive Bayesian classification. J Mach Learn Res 87(1):93–125

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing, University of Kent, Canterbury, CT2 7NF, UK

    Khalid M. Salama & Alex A. Freitas

Authors
  1. Khalid M. Salama
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alex A. Freitas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Khalid M. Salama.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Salama, K.M., Freitas, A.A. ABC-Miner+: constructing Markov blanket classifiers with ant colony algorithms. Memetic Comp. 6, 183–206 (2014). https://doi.org/10.1007/s12293-014-0138-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12293-014-0138-6

Keywords

Search

Navigation