Abstract
Kernel-based clustering is one of the most popular methods for partitioning nonlinearly separable datasets. However, exhaustive search for the global optimum is NP-hard. Iterative procedure such as k-means can be used to seek one of the local minima. Unfortunately, it is easily trapped into degenerate local minima when the prototypes of clusters are ill-initialized. In this paper, we restate the optimization problem of kernel-based clustering in an online learning framework, whereby a conscience mechanism is easily integrated to tackle the ill-initialization problem and faster convergence rate is achieved. Thus, we propose a novel approach termed conscience online learning (COLL). For each randomly taken data point, our method selects the winning prototype based on the conscience mechanism to bias the ill-initialized prototype to avoid degenerate local minima and efficiently updates the winner by the online learning rule. Therefore, it can more efficiently obtain smaller distortion error than k-means with the same initialization. The rationale of the proposed COLL method is experimentally analyzed. Then, we apply the COLL method to the applications of digit clustering and video clustering. The experimental results demonstrate the significant improvement over existing kernel-based clustering methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abolhassani B, Salt JE, Dodds DE (2004) A two-phase genetic k-means algorithm for placement of radioports in cellular networks. IEEE Trans Syst Man Cybern B Cybern 34: 533–538
Asuncion A, Newman D (2007) UCI machine learning repository. http://www.ics.uci.edu/~mlearn/MLRepository.html
Bishop CM (2006) Pattern recognition and machine learning. Springer, Berlin
Bradley PS, Fayyad UM (1998) Refining initial points for k-means clustering of the 15th international conference on machine learning
Cheung Y-M (2005) On rival penalization controlled competitive learning for clustering with automatic cluster number selection. IEEE Trans Knowl Data Eng 17: 1583–1588
Denton AM, Besemann CA, Dorr DH (2009) Pattern-based time-series subsequence clustering using radial distribution functions. Knowl Inf Syst 18: 1–27
DeSieno D (1988) Adding a conscience to competitive learning. IEEE international conference on neural network
Dhillon IS, Guan Y, Kulis B (2004) Kernel k-means, spectral clustering and normalized clustering of the 10th ACM SIGKDD international conference on knowledge discovery and data mining
http://www.open-video.org (n.d.) The Open Video Project is managed at the Interaction Design Laboratory, at the School of Information and Library Science, University of North Carolina at Chapel Hill
Hubert L, Arabie P (1985) Comparing partitions. J Classif 2: 193–218
Hull JJ (1994) A database for handwritten text recognition research. IEEE Trans Pattern Anal Mach Intell 16(5): 550–554
Jin R, Goswami A, Agrawal G (2006) Fast and exact out-of-core and distributed k-means clustering. Knowl Inf Syst 10: 17–40
Jing L, Ng MK, Huang JZ (2010) Knowledge-based vector space model for text clustering. Knowl Inf Syst 25: 35–55
Khan SS, Ahmad A (2004) Cluster center initialization algorithm for k-means clustering. Pattern Recognit Lett 25: 1293–1302
Krishna K, Murty MN (1999) Genetic k-means algorithm. IEEE Trans Syst Man Cybern B Cybern 29(3): 433–439
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. http://yann.lecun.com/exdb/mnist/
Likas A, Vlassis N, Verbeek JJ (2003) The global k-means clustering algorithm. Pattern Recognit 36: 451–461
Liu B, Xia Y, Yu PS (2000) Clustering through decision tree construction. In: Proceedings of the 9th international conference on information and knowledge management
MacQueen J (1967) Some methods for classification and analysis of multivariate observations. In: Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, vol 1. University of California Press, California, pp 281–297
Nayak R (2008) Fast and effective clustering of XML data using structural information. Knowl Inf Syst 14: 197–215
Schölkopf B (2000) The kernel trick for distances. Adv Neural Inf Process Syst 301–307
Schölkopf B, Smola A, Müller K-R (1998) Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput 10: 1299–1319
Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, Cambridge
Strehl A, Ghosh J (2002) Cluster ensembles—a knowledge reuse framework for combining multiple partitions. J Mach Learn Res 3: 583–617
Strehl A, Ghosh J, Mooney RJ (2000) Impact of similarity measures on web-page clustering. In: Proceedings of the AAAI workshop on AI for web search (AAAI 2000)
Su Z, Yang Q, Zhang H, Xu X, Hu Y-H, Ma S (2002) Correlation-based web document clustering for adaptive web interface design. Knowl Inf Syst 4: 151–167
Takacs B, Demiris Y (2010) Spectral clustering in multi-agent systems. Knowl Inf Syst 25: 607–622
Truong BT, Venkatesh S (2007) Video abstraction: a systematic review and classification. ACM Trans Multime Comput Commun Appl 3(1): 1–37
Tzortzis GF, Likas AC (2009) The global kernel k-means algorithms for clustering in feature space. IEEE Trans Neural Netw 20(7): 1181–1194
Wang C-D, Lai J-H (2011) Energy based competitive learning. Neurocomputing 74: 2265–2275
Wang C-D, Lai J-H, Zhu J-Y (2010) A conscience on-line learning approach for kernel-based clustering. In: Proceedings of the 10th international conference on data mining. pp 531–540
Wang J, Wu X, Zhang C (2005) Support vector machines based on k-means clustering for real-time business intelligence systems. Int J Bus Intell Data Min 1: 54–64
Wang K, Xu C, Liu B (1999) Clustering transactions using large items. In: Proceedings of the 8th international conference on information and knowledge management
Wu J, Xiong H, Chen J (2009) Adapting the right measures for k-means clustering. In: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining
Wu J, Xiong H, Chen J, Zhou W (2007) A generalization of proximity functions for k-means. In: Proceedings of the 7th international conference on data mining
Wu X, Kumar V, Quinlan JR, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS, Zhou Z-H, Steinbach M, Hand DJ, Steinberg D (2008) Top 10 algorithms in data mining. Knowl Inf Syst 14: 1–37
Xiong H, Steinbach M, Ruslim A, Kumar V (2009) Characterizing pattern preserving clustering. Knowl Inf Syst 19: 311–336
Xu L, Krzyżak A, Oja E (1993) Rival penalized competitive learning for clustering analysis, rbf net, and curve detection. IEEE Trans Neural Netw 4(4): 636–649
Xu R, Wunsch DI (2005) Survey of clustering algorithms. IEEE Trans Neural Netw 16(3): 645–678
Zhang Y-J, Liu Z-Q (2002) Self-splitting competitive learning: a new on-line clustering paradigm. IEEE Trans Neural Netw 13(2): 369–380
Zhang Z, Dai BT, Tung AK (2006) On the lower bound of local optimums in k-means algorithm. In: Proceedings of the 6th international conference on data mining
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, CD., Lai, JH. & Zhu, JY. Conscience online learning: an efficient approach for robust kernel-based clustering. Knowl Inf Syst 31, 79–104 (2012). https://doi.org/10.1007/s10115-011-0416-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-011-0416-2