Abstract
Unsupervised learning is very important in the processing of multimedia content as clustering or partitioning of data in the absence of class labels is often a requirement. This chapter begins with a review of the classic clustering techniques of k-means clustering and hierarchical clustering. Modern advances in clustering are covered with an analysis of kernel-based clustering and spectral clustering. One of the most popular unsupervised learning techniques for processing multimedia content is the self-organizing map, so a review of self-organizing maps and variants is presented in this chapter. The absence of class labels in unsupervised learning makes the question of evaluation and cluster quality assessment more complicated than in supervised learning. So this chapter also includes a comprehensive analysis of cluster validity assessment techniques.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
M. A. Aizerman, E. M. Braverman, and L. I. Rozonoer. Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control, 25(6):821–837, 1964.
A. Ben-Hur, A. Elisseeff, and I. Guyon. A stability based method for discovering structure in clustered data. In Proceedings of the 7th Pacific Symposium on Biocomputing (PSB 2002), pp. 6–17, Lihue, HI, January 2002.
J. C. Bezdek and N. R. Pal. Cluster validation with generalized dunn’s indices. In ANNES ’95: Proceedings of the 2nd New Zealand Two-Stream International Conference on Artificial Neural Networks and Expert Systems, p. 190, Washington, DC, USA, 1995. IEEE Computer Society.
J. Blackmore and R. Miikkulainen. Incremental grid growing: encoding high-dimensional structure into a two-dimensional feature map. In Proceedings of the ICNN’93, International Conference on Neural Networks, Vol. I, pp. 450–455, Piscataway, NJ, 1993. IEEE Service Center.
N. Bolshakova and F. Azuaje. Cluster validation techniques for genome expression data. Technical Report TCD-CS-2002-33, Trinity College Dublin, September 2002.
M. Brand and K. Huang. A unifying theorem for spectral embedding and clustering. In Proceedings of the 9th International Workshop on AI and Statistics, January 2003.
T. Calinski and J. Harabasz. A dendrite method for cluster analysis. Communications in Statistics, 3:1–27, 1974.
N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines: and Other Kernel-Based Learning Methods. Cambridge University Press, New York, NY, USA, 2000.
D. L. Davies and W. Bouldin. A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1(2):224–227, 1979.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, 39:1–38, 1977.
I. S. Dhillon. Co-clustering documents and words using bipartite spectral graph partitioning. In Knowledge Discovery and Data Mining, pp. 269–274, 2001.
I. S. Dhillon, Y. Guan, and B. Kulis. Kernel k-means: spectral clustering and normalized cuts. In Proceedings of the 2004 ACM SIGKDD International conference on Knowledge Discovery and Data Mining, pp. 551–556. New York, NY, 2004. ACM Press.
C. Ding and X. He. Cluster merging and splitting in hierarchical clustering algorithms. In Proceedings of the 2002 IEEE International Conference on Data Mining (ICDM’02), p. 139. Washington, DC, 2002. IEEE Computer Society.
W. E. Donath and A. J. Hoffman. Lower bounds for the partitioning of graphs. IBM Journal of Research and Development, 17:420–425, 1973.
R. C. Dubes. How many clusters are best? – an experiment. Pattern Recognition, 20(6):645–663, 1987.
J. C. Dunn. A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters. Journal of Cybernetics, 3:32–57, 1974.
J. C. Dunn. Well separated clusters and optimal fuzzy-partitions. Journal of Cybernetics, 4:95–104, 1974.
M. Fiedler. Algebraic connectivity of graphs. Czechoslovak Mathematical Journal, 23(98):298–305, 1973.
B. Fischer and J. M. Buhmann. Path-based clustering for grouping of smooth curves and texture segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions, 25(4):513–518, April 2003.
E. W. Forgy. Cluster analysis of multivariate data: efficiency vs interpretability of classifications. Biometrics, 21:768–769, 1965.
E. B. Fowlkes and C. L. Mallow. A method for comparing two hierarchical clusterings. Journal of American Statistical Association, 78:553–569, 1983.
B. Fritzke. Growing cell structures—a self-organizing network in k dimensions. In I. Aleksander and J. Taylor, editors, Artificial Neural Networks, 2, Vol. II, pp. 1051–1056, Amsterdam, Netherlands, 1992. North-Holland.
B. Fritzke. Growing grid – a self-organizing network with constant neighborhood range and adaptation strength. Neural Processing Letters, 2(5):9–13, 1995.
J. Ghosh. Scalable clustering methods for data mining. In N. Ye, editor, Handbook of Data Mining, chapter 10. Mahwah, NJ, 2003. Lawrence Erlbaum.
C. D. Giurcaneanu and I. Tabus. Cluster structure inference based on clustering stability with applications to microarray data analysis. EURASIP Journal on Applied Signal Processing, 1:64–80, 2004.
S. Guha, R. Rastogi, and K. Shim. CURE: an efficient clustering algorithm for large databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 73–84, 1998.
K. M. Hall. An r-dimensional quadratic placement algorithm. Management Science, 17(3):219–229, November 1970.
V. Hautamäki, S. Cherednichenko, I. Kärkkäinen, T. Kinnunen, and P. Fränti. Improving k-means by outlier removal. In Image Analysis, 14th Scandinavian Conference, SCIA 2005, pp. 978–987, 2005.
L. J. Hubert and P. Arabie. Comparing partitions. Journal of Classification, 2:193–218, 1985.
L. J. Hubert and J. R. Levin. A general statistical framework for accessing categorical. Psychological Bulletin, 83:1072–1082, 1976.
P. Jaccard. The distribution of flora in the alpine zone. New Phytologist, 11(2):37–50, 1912.
S. Kaski, J. Kangas, and T. Kohonen. Bibliography of self-organizing map (SOM) papers 1981–1997. Neural Computing Surveys, 1(3&4):1–176, 1998.
B. W. Kernighan and S. Lin. An efficient heuristic procedure for partitioning graphs. The Bell System Technical Journal, 49(2):291–308, 1970.
Y. Kluger, R. Basri, J. T. Chang, and M. Gerstein. Spectral biclustering of microarray data: coclustering genes and conditions. Genome Research, 13:703–716, April 2003.
T. Kohonen. Self-Organizing Maps. Springer-Verlag, New York, NY, 2001.
T. Kohonen, E. Oja, O. Simula, A. Visa, and J. Kangas. Engineering applications of the self-organizing map. Proceedings of the IEEE, 84(10):1358–1384, October 1996.
T. Lange, V. Roth, M. L. Braun, and J. M. Buhmann. Stability-based validation of clustering solutions. Neural Computation, 16(6):1299–1323, 2004.
B. Larsen and C. Aone. Fast and effective text mining using linear-time document clustering. In KDD ’99: Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 16–22, New York, NY, USA, 1999. ACM Press.
M. Law and A. K. Jain. Cluster validity by bootstrapping partitions. Technical Report MSU-CSE-03-5, University of Washington, February 2003.
E. Levine and E. Domany. Resampling method for unsupervised estimation of cluster validity. Neural Computation, 13(11):2573–2593, 2001.
M. Meila. Comparing clusterings. Technical Report 418, University of Washington, 2002.
D. Merkl. Exploration of text collections with hierarchical feature maps. In Research and Development in Information Retrieval, pp. 186–195, 1997.
R. Miikkulainen. Script recognition with hierarchical feature maps. Connection Science, 2(1&2):83–101, 1990.
G. W. Milligan and M. C. Cooper. An examination of procedures for determining the number of clusters in a data set. Psychometrika, 50(2):159–179, 1985.
A. Ng, M. Jordan, and Y. Weiss. On spectral clustering: analysis and an algorithm. In Proceedings of the Advances in Neural Information Processing, 2001.
M. Oja, S. Kaski, and T. Kohonen. Bibliography of self-organizing map (SOM) papers: 1998–2001 addendum. Neural Computing Surveys, 3:1–156, 2003.
A. Pothen, H. D. Simon, and K.-P. Liou. Partitioning sparse matrices with eigenvectors of graphs. SIAM Journal of Mathematical Analysis and Applications, 11(3):430–452, 1990.
W. M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(66):846–850, 1971.
A. Rauber and D. Merkl. The SOMLib digital library system. In Proceedings of the 3rd European Conference on Research and Advanced Technology for Digital Libraries (ECDL’99), Lecture Notes in Computer Science (LNCS 1696), pp. 323–342, Paris, France, September 22-24 1999. Springer.
A. Rauber, D. Merkl, and M. Dittenbach. The growing hierarchical self-organizing map: Exploratory analysis of high-dimensional data. IEEE Transactions on Neural Networks, 13(6):1331–1341, November 2002.
P. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20(1):53–65, 1987.
J. W. Sammon Jr. A nonlinear mapping for data structure analysis. IEEE Transactions on Computers, C-18(5):401–409, May 1969.
B. Schölkopf, A. Smola, and K-R. Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5):1299–1319, 1998.
J. Shi and J. Malik. Normalized cuts and image segmentation. In Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR ’97), pp. 731–737. Huntsville, AL, 1997. IEEE Computer Society.
J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 22(8):888–905, August 2000.
M. Steinbach, G. Karypis, and V. Kumar. A comparison of document clustering techniques. In Proceedings of KDD Workshop on Text Mining 2000, 2000.
A. Strehl and J. Ghosh. Cluster ensembles – a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research, 3:583–617, December 2002.
R. Tibshirani, G. Walther, D. Botstein, and P. Brown. Cluster validation by prediction strength. Technical report, Statistics Department, Stanford University, 2001.
D. Verma and M. Meila. A comparison of spectral clustering algorithms. Technical report, University of Washington, 2003.
S. X. Yu and J. Shi. Multiclass spectral clustering. In Proceedings of the 9th IEEE International Conference on Computer Vision, p. 313, October 2003.
T. Zhang, R. Ramakrishnan, and M. Livny. BIRCH: an efficient data clustering method for very large databases. Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, pp. 103–114, 1996.
Y. Zhao and G. Karypis. Criterion functions for document clustering: experiments and analysis. Technical Report 01-040, University of Minnesota, November 2001.
Y. Zhao and G. Karypis. Evaluation of hierarchical clustering algorithms for document datasets. In Proceedings of the Eleventh International Conference on Information and Knowledge Management, pp. 515–524. New York, NY, 2002. ACM Press.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Greene, D., Cunningham, P., Mayer, R. (2008). Unsupervised Learning and Clustering. In: Cord, M., Cunningham, P. (eds) Machine Learning Techniques for Multimedia. Cognitive Technologies. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-75171-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-540-75171-7_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-75170-0
Online ISBN: 978-3-540-75171-7
eBook Packages: Computer ScienceComputer Science (R0)