Skip to main content
SpringerLink
Log in

Self-paced and Bayes-decision-rule linear KNN prediction

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

While a testing sample may be first encoded linearly with labeled samples and then classified with KNN on the sum of the obtained weights of the samples in each class so as to avoid the consistent distribution assumption explicitly or implicitly used in most of the existing classification methods for training and testing samples, a novel self-paced and Bayes-decision-rule linear KNN prediction method SBLD-KNN in this study will be proposed to achieve three goals: (1) class-ware information will be explicitly reflected in a grouping effect regularization term so as to share the sparsity of a linear encoder and simultaneously have grouping effect of weights on each class; (2) the resultant predictor behaves like Bayes-decision-rule for minimum error; (3) self-paced regularized term is designed to adaptively truncate the weights of labeled samples for enhancing generalization. In order to do so, the corresponding objective function of SBLD-KNN is designed and then optimized by using the alternating optimization strategy, and its Bayes-decision-rule is theoretically analyzed. Our experimental results on benchmark datasets witness the effectiveness of SBLD-KNN, in contrast to the comparative methods, including SBLD-KNN’s simplified version BD-KNN with weight’s truncating rather than self-pacing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig.1
Fig.2
Fig.3

Similar content being viewed by others

References

  1. Pourpanah F, Wang R, Wang X (2019) Feature selection for data classification based on binary brain storm optimization. In: 2019 IEEE 6th international conference on cloud computing and intelligence systems (CCIS), 19–21 Dec. 2019, pp 108–113. https://doi.org/10.1109/CCIS48116.2019.9073751

  2. Chan PP, Wang Y, Yeung DS (2020) Adversarial attack against deep reinforcement learning with static reward impact map. In: Proceedings of the 15th ACM Asia conference on computer and communications security, pp 334–343

  3. Mo D, Lai Z, Wang X, Wong W (2020) Jointly sparse locality regression for image feature extraction. IEEE Trans Multimed 22(11):2873–2888. https://doi.org/10.1109/TMM.2019.2961508

    Article  Google Scholar 

  4. Chan PPK et al (2018) Face liveness detection using a flash against 2D spoofing attack. IEEE Trans Inf Forensics Secur 13(2):521–534. https://doi.org/10.1109/TIFS.2017.2758748

    Article  Google Scholar 

  5. Lian G, Wang Y, Qin H, Chen G (2021) Towards unified on-road object detection and depth estimation from a single image. Int J Mach Learn Cybern. https://doi.org/10.1007/s13042-021-01444-z

    Article  Google Scholar 

  6. De A, Guo C (2014) An image segmentation method based on the fusion of vector quantization and edge detection with applications to medical image processing. Int J Mach Learn Cybern 5(4):543–551. https://doi.org/10.1007/s13042-013-0205-1

    Article  Google Scholar 

  7. Premachandra C, Premachandra HWH, Parape CD, Kawanaka H (2015) Road crack detection using color variance distribution and discriminant analysis for approaching smooth vehicle movement on non-smooth roads. Int J Mach Learn Cybern 6(4):545–553. https://doi.org/10.1007/s13042-014-0240-6

    Article  Google Scholar 

  8. Zhang J, Liu D, Ma Y (2021) Delay-dependent stability analysis of nonlinear truck-trailer systems based on non-fragile memory sampled-data via fuzzy control. Int J Mach Learn Cybern 12(12):3475–3490. https://doi.org/10.1007/s13042-021-01386-6

    Article  Google Scholar 

  9. Zheng W, Liu H, Wang B, Sun F (2020) Cross-modal learning for material perception using deep extreme learning machine. Int J Mach Learn Cybern 11(4):813–823. https://doi.org/10.1007/s13042-019-00962-1

    Article  Google Scholar 

  10. Zhao L, Zhu J (2019) Learning from correlation with extreme learning machine. Int J Mach Learn Cybern 10(12):3635–3645. https://doi.org/10.1007/s13042-019-00949-y

    Article  Google Scholar 

  11. Zhu H, Wang X, Wang R (2021) Fuzzy monotonic K-nearest neighbor versus monotonic fuzzy K-nearest neighbor. IEEE Trans Fuzzy Syst. https://doi.org/10.1109/TFUZZ.2021.3117450

    Article  Google Scholar 

  12. France S, Carroll D (2009) Is the distance compression effect overstated? Some theory and experimentation. In: Perner P (ed) Machine learning and data mining in pattern recognition. Springer, Berlin, pp 280–294

    Chapter  Google Scholar 

  13. Bian Z, Ishibuchi H, Wang S (2019) Joint learning of spectral clustering structure and fuzzy similarity matrix of data. IEEE Trans Fuzzy Syst 27(1):31–44. https://doi.org/10.1109/TFUZZ.2018.2856081

    Article  Google Scholar 

  14. Bian Z, Vong CM, Wong PK, Wang S (2020) Fuzzy KNN method with adaptive nearest neighbors. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2020.3031610

    Article  Google Scholar 

  15. Shan H, Zhang J, Kruger U (2015) Learning linear representation of space partitioning trees based on unsupervised kernel dimension reduction. IEEE Trans Cybern 46(12):3427–3438

    Article  Google Scholar 

  16. Wang J, Yang J, Yu K, Lv F, Huang T, Gong Y (2010) Locality-constrained linear coding for image classification. In: 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 3360–3367

  17. Zhong LW, Kwok JT (2012) Efficient sparse modeling with automatic feature grouping. IEEE Trans Neural Netw Learn Syst 23(9):1436–1447

    Article  Google Scholar 

  18. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J Roy Stat Soc Ser B (Methodol) 58(1):267–288

    MathSciNet  MATH  Google Scholar 

  19. Liu Q, Liu C (2016) A novel locally linear KNN method with applications to visual recognition. IEEE Trans Neural Netw Learn Syst 28(9):2010–2021

    Article  MathSciNet  Google Scholar 

  20. Jiang L, Meng D, Yu S-I, Lan Z, Shan S, Hauptmann A (2014) Self-paced learning with diversity. Adv Neural Inf Process Syst 27:2078–2086

    Google Scholar 

  21. Kumar MP, Packer B, Koller, D (2010) Self-paced learning for latent variable models. In: Lafferty J, Williams CKI, Shawe-Taylor J, Zemel RS, Culotta A (eds) Advances in neural information processing systems. vol 23, pp 1189–1197

  22. Jiang L, Meng D, Mitamura T, Hauptmann AG (2014) Easy samples first: Self-paced reranking for zero-example multimedia search. In: Proceedings of the 22nd ACM international conference on multimedia, pp 547–556

  23. Supancic JS, Ramanan D (2013) Self-paced learning for long-term tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2379–2386

  24. Fukunaga K (2013) Introduction to statistical pattern recognition. Elsevier, Amsterdam

    MATH  Google Scholar 

  25. Dong A, Chung F-L, Deng Z, Wang S (2015) Semi-supervised SVM with extended hidden features. IEEE Trans Cybern 46(12):2924–2937

    Article  Google Scholar 

  26. Jiang Y, Chung F-L, Ishibuchi H, Deng Z, Wang S (2014) Multitask TSK fuzzy system modeling by mining intertask common hidden structure. IEEE Trans Cybern 45(3):534–547

    Article  Google Scholar 

  27. Deng Z, Jiang Y, Chung F-L, Ishibuchi H, Wang S (2012) Knowledge-leverage-based fuzzy system and its modeling. IEEE Trans Fuzzy Syst 21(4):597–609

    Article  Google Scholar 

  28. Wright SJ (2015) Coordinate descent algorithms. Math Progr 151(1):3–34

    Article  MathSciNet  Google Scholar 

  29. Zhang M-L, Zhou Z-H (2007) ML-KNN: A lazy learning approach to multi-label learning. Pattern Recogn 40(7):2038–2048

    Article  Google Scholar 

  30. Frank A, Asuncion A (2010) UCI machine learning repository, 2010. [Online]. Available: http://archive.ics.uci.edu/ml

  31. Alcalá-Fdez J, Fernández A, Luengo J et al (2011) Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. J Multiple-Valued Logic Soft Comput 17:255–287

    Google Scholar 

  32. Hui Z, Hastie T (2005) Regularization and variable selection via the elastic net. J Roy Stat Soc 67(5):768–768

    Article  MathSciNet  Google Scholar 

  33. Kollios G, Gunopulos D, Koudas N, Berchtold S (2003) Efficient biased sampling for approximate clustering and outlier detection in large data sets. Knowl Data Eng IEEE Trans 15(5):1170–1187

    Article  Google Scholar 

  34. Scott DW (2015) Multivariate density estimation: theory, practice, and visualization[M]. Wiley

  35. Fawcett T (2006) An introduction to ROC analysis. Pattern Recogn Lett 27(8):861–874

    Article  MathSciNet  Google Scholar 

  36. Hao H, Wang J, Abudureyimu H (2012) Maximum F1-score discriminative training for automatic mispronunciation detection in computer-assisted language learning. In: Interspeech 2012, pp 815–818

  37. Gu S, Chung F-L, Wang S (2019) A novel deep fuzzy classifier by stacking adversarial interpretable TSK fuzzy sub-classifiers with smooth gradient information. IEEE Trans Fuzzy Syst 28(7):1369–1382

    Google Scholar 

  38. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  39. Bertini JR Jr, Zhao L, Motta R, de Andrade Lopes A (2011) A nonparametric classification method based on k-associated graphs. Inf Sci 181(24):5435–5456

    Article  MathSciNet  Google Scholar 

  40. Gönen M, Alpaydın E (2011) Multiple kernel learning algorithms J. Mach Learn Res 12:2211–2268

    MathSciNet  MATH  Google Scholar 

  41. Zhang T, Deng Z, Wu D, Wang S (2018) Multiview fuzzy logic system with the cooperation between visible and hidden views. IEEE Trans Fuzzy Syst 27(6):1162–1173

    Article  Google Scholar 

  42. Mathew J, Pang CK, Luo M et al (2017) Classification of imbalanced data by oversampling in kernel space of support vector machines. IEEE Trans Neural Netw Learn Syst 29(9):4065–4076

    Article  Google Scholar 

  43. Zar JH (1999) Biostatistical analysis. Pearson Education India, Noida

    Google Scholar 

  44. Nash MS (2001) Handbook of parametric and nonparametric statistical procedures. Technometrics 43(3):374

    Article  Google Scholar 

  45. Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. Siam J Imaging Sci 2(1):183–202

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants U20A20228, 61772198, 6197071117, by the Natural Science Foundation of Jiangsu Province under Grant BK20191331, and by National First-class Discipline Program of Light Industry and Engineering (LITE2018). (Corresponding author: Shitong Wang). (Jin Zhang and Zekang Bian have equal contribution to this work).

Author information

Authors and Affiliations

  1. School of AI and Computer Science, Jiangnan University, Wuxi, Jiangsu, China

    Jin Zhang, Zekang Bian & Shitong Wang

  2. Taihu Jiangsu Key Construction Lab. of IoT Application Technologies, Wuxi, China

    Jin Zhang, Zekang Bian & Shitong Wang

Authors
  1. Jin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zekang Bian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shitong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shitong Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Bian, Z. & Wang, S. Self-paced and Bayes-decision-rule linear KNN prediction. Int. J. Mach. Learn. & Cyber. 13, 3267–3283 (2022). https://doi.org/10.1007/s13042-022-01593-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01593-9

Keywords

Search

Navigation