Skip to main content
SpringerLink
Log in

Two improved attribute weighting schemes for value difference metric

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Due to its simplicity, efficiency and efficacy, value difference metric (VDM) has continued to perform well against more sophisticated newcomers and thus has remained of great interest to the distance metric learning community. Of numerous approaches to improving VDM by weakening its attribute independence assumption, attribute weighting has received less attention (only two attribute weighting schemes) but demonstrated remarkable class probability estimation performance. Among two existing attribute weighting schemes, one is non-symmetric and the other is symmetric. In this paper, we propose two simple improvements for setting attribute weights for use with VDM. One is the non-symmetric Kullback–Leibler divergence weighted value difference metric (KLD-VDM) and the other is the symmetric gain ratio weighted value difference metric (GR-VDM). We performed extensive evaluations on a large number of datasets and found that KLD-VDM and GR-VDM significantly outperform two existing attribute weighting schemes in terms of the negative conditional log likelihood and root relative squared error, yet at the same time maintain the computational simplicity and robustness that characterize VDM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Alcalá-Fdez J, Fernandez A, Luengo J, Derrac J, García S, Sánchez L, Herrera F (2011) KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. J Mult Valued Log Soft Comput 17(2–3):255–287

    Google Scholar 

  2. Bian W, Tao D (2012) Constrained empirical risk minimization framework for distance metric learning. IEEE Trans Neural Netw Learn Syst 23(8):1194–1205

    Article  Google Scholar 

  3. Blanzieri E, Ricci F (1999) Probability based metrics for nearest neighbor classification and case-based reasoning. In: Proceedings of the 3rd International conference on case-based reasoning. Springer, pp 14–28

  4. Cost S, Salzberg S (1993) A weighted nearest neighbor algorithm for learning with symbolic features. Mach Learn 10:57–78

    Google Scholar 

  5. Davis JV, Kulis B, Jain P, Sra S, Dhillon IS (2007) Information-theoretic metric learning. In: Proceedings of the 24th conference on machine learning. ACM Press, Corvalis, pp 209–216

  6. Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  7. Domeniconi C, Gunopulos D (2001) Adaptive nearest neighbor classification using support vector machines. In: Advances in neural information processing systems 14. MIT Press, Cambridge, pp 665–672

  8. Domeniconi C, Peng J, Gunopulos D (2000) Adaptive metric nearest-neighbor classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Hilton Head, p 1517

  9. Domingos P, Pazzani M (1997) On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 29:103–130

    Article  MATH  Google Scholar 

  10. Elkan C (2001) The foundations of cost-sensitive learning. In: Proceedings of the 17th international joint conference on artificial intelligence, pp 973–978

  11. Frank A, Asuncion A (2010) UCI machine learning repository. University of California, Irvine

    Google Scholar 

  12. Garcia S, Herrera F (2008) An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons. J Mach Learn Res 9:2677–2694

    MATH  Google Scholar 

  13. Grossman D, Domingos P (2004) Learning Bayesian network classifiers by maximizing conditional likelihood. In: Proceedings of the 21st international conference on machine learning. ACM, pp 361–368

  14. Guo Y, Greiner R (2005) Discriminative model selection for belief net structures. In: Proceedings of the 12th national conference on artificial intelligence. AAAI, pp 770–776

  15. Hall M (2007) A decision tree-based attribute weighting filter for naive Bayes. Knowl Based Syst 20:120–126

    Article  Google Scholar 

  16. Hall MA (2000) Correlation-based feature selection for discrete and numeric class machine learning. In: Proceedings of the 17th international conference on machine learning. Morgan Kaufmann, Stanford, pp 359–366

  17. Hastie T, Tibshirani R (1996) Discriminant adaptive nearest neighbor classification. IEEE Trans Pattern Anal Mach Intell 18(6):607–616

    Article  Google Scholar 

  18. Hu JH, Zhan DC, Wu X, Jiang Y, Zhou ZH (2015) Pairwised specific distance learning from physical linkages. ACM Trans Knowl Discov Data 9(3):20

    Article  Google Scholar 

  19. Jiang L, Li C (2013) An augmented value difference measure. Pattern Recognit Lett 34(10):1169–1174

    Article  Google Scholar 

  20. Jiang L, Li C, Zhang H, Cai Z (2014) A novel distance function: frequency difference metric. Int J Pattern Recognit Artif Intell 28(2):1451002

    Article  Google Scholar 

  21. Kasif S, Salzberg S, Waltz D, Rachlin J, Aha D (1998) A probabilistic framework for memory-based reasoning. Artif Intell 104:287–311

    Article  MathSciNet  MATH  Google Scholar 

  22. Kira K, Rendell L (1992) A practical approach to feature selection. In: Proceedings of the 9th international conference on machine learning. Morgan Kaufman, Aberdeen, pp 249–256

  23. Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86

    Article  MathSciNet  MATH  Google Scholar 

  24. Lee CH, Gutierrez F, Dou D (2011) Calculating feature weights in naive Bayes with Kullback–Leibler measure. In: Proceedings of the 11th IEEE international conference on data mining. IEEE, Vancouver, pp 1146-1151

  25. Li C, Jiang L, Li H (2014) Local value difference metric. Pattern Recognit Lett 49:62–68

    Article  Google Scholar 

  26. Li C, Jiang L, Li H (2014) Naive Bayes for value difference metric. Front Comput Sci 8(2):255–264

    Article  MathSciNet  Google Scholar 

  27. Li C, Jiang L, Li H, Wu J, Zhang P (2017) Toward value difference metric with attribute weighting. Knowl Inf Syst 50(3):795–825

    Article  Google Scholar 

  28. Li C, Li H (2011) One dependence value difference metric. Knowl Based Syst 24(5):589–594

    Article  Google Scholar 

  29. Li C, Li H (2012) A modified short and fukunaga metric based on the attribute independence assumption. Pattern Recognit Lett 33(9):1213–1218

    Article  Google Scholar 

  30. Li C, Li H (2013) Selective value difference metric. J Comput 8(9):2232–2238

    Google Scholar 

  31. Mitchell TM (1997) Machine learning, 1st edn. McGraw-Hill, New York City

    MATH  Google Scholar 

  32. Myles JP, Hand DJ (1990) The multi-class metric problem in nearest neighbour discrimination rules. Pattern Recognit 23(11):1291–1297

    Article  Google Scholar 

  33. Nadeau C, Bengio Y (2003) Inference for the generalization error. Mach Learn 52(3):239–281

    Article  MATH  Google Scholar 

  34. Qiu C, Jiang L, Li C (2015) Not always simple classification: learning superparent for class probability estimation. Expert Syst Appl 42(13):5433–5440

    Article  Google Scholar 

  35. Quinlan JR (1986) Induction of decision trees. Mach Learn 1:81–106

    Google Scholar 

  36. Quinlan JR (1993) C4.5: programs for machine learning, 1st edn. Morgan Kaufmann, San Mateo

    Google Scholar 

  37. Robnik-Sikonja M, Kononenko I (2003) Theoretical and empirical analysis of relieff and rrelieff. Mach Learn 53(1–2):23–69

    Article  MATH  Google Scholar 

  38. Saar-Tsechansky M, Provost F (2004) Active sampling for class probability estimation and ranking. Mach Learn 54:153–178

    Article  MATH  Google Scholar 

  39. Sangineto E (2013) Pose and expression independent facial landmark localization using dense-surf and the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell 35(3):624–638

    Article  Google Scholar 

  40. Short RD, Fukunaga K (1981) The optimal distance measure for nearest neighbour classification. IEEE Trans Inf Theory 27:622–627

    Article  MATH  Google Scholar 

  41. Stanfill C, Waltz D (1986) Toward memory-based reasoning. Commun ACM 29:1213–1228

    Article  Google Scholar 

  42. Weinberger KQ, Blitzer JC, Saul LK (2006) Distance metric learning for large margin nearest neighbor classification. Adv Neural Inf Process Syst 18:1473–1480

    Google Scholar 

  43. Wilson DR, Martinez TR (1997) Improved heterogeneous distance functions. J Artif Intell Res 6:1–34

    Article  MathSciNet  MATH  Google Scholar 

  44. Witten IH, Frank E, Hall MA (2011) Data mining: practical machine learning tools and techniques, 3rd edn. Morgan Kaufmann, San Francisco

    Google Scholar 

  45. Yang L (2006) Distance metric learning: a comprehensive survey. Technical report, Department of Computer Science and Engineering, Michigan State University

  46. Zaidi NA, Cerquides J, Carman MJ, Webb GI (2013) Alleviating naive Bayes attribute independence assumption by attribute weighting. J Mach Learn Res 14:1947–1988

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The work was partially supported by the National Natural Science Foundation of China (U1711267), the Program for New Century Excellent Talents in University (NCET-12-0953), the Chenguang Program of Science and Technology of Wuhan (2015070404010202), and the Open Research Project of Hubei Key Laboratory of Intelligent Geo-Information Processing (KLIGIP201601).

Author information

Authors and Affiliations

  1. Department of Computer Science, China University of Geosciences, Wuhan, 430074, China

    Liangxiao Jiang

  2. Department of Mathematics, China University of Geosciences, Wuhan, 430074, China

    Chaoqun Li

Authors
  1. Liangxiao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chaoqun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liangxiao Jiang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, L., Li, C. Two improved attribute weighting schemes for value difference metric. Knowl Inf Syst 60, 949–970 (2019). https://doi.org/10.1007/s10115-018-1229-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-018-1229-3

Keywords

Search

Navigation