Abstract
Scalability and high dimensionality are two common problems associated with document clustering. We present a novel scheme to deal with these problems. Given a set of documents, we partition the set into several parts. We use one part and cluster the constituent documents into groups. By the obtained groups, we reduce the number of features by a certain ratio. Then we add another part, cluster the documents into groups based on the reduced features, and further reduce the number of the remaining features. This process is iterated until all parts are used. Experimental results have shown that our proposed scheme is effective for clustering large high-dimensional document datasets.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Dhillon, I.S., Guan, Y., Fan, J.: Efficient Clustering of Very Large Document Collections. In: Data Mining for Scientific and Engineering Applications, pp. 357–381. Kluwer Academic Publishers, Dordrecht (2001)
Dhillon, I.S., Kogan, J., Nicholas, M.: Feature Selection and Document Clustering. In: A Comprehensive Survey of Text Mining, pp. 73–100. Springer, Heidelberg (2003)
Kogan, J., Teboulle, M., Nicholas, C.: Data Driven Similarity Measures for K-Means Like Clustering Algorithms. Information Retrieval 8(2), 331–349 (2005)
Boley, D.: Principal Direction Divisive Partitioning. Data Mining and Knowledge Discovery 2(4), 325–344 (1998)
Salton, G., McGill, M.J.: Introduction to Modern Retrieval. McGraw-Hill Book Company, New York (1983)
Dhillon, I.S., Modha, D.S.: Concept Decompositions for Large Sparse Text Data using Clustering. Machine Learning 42(1), 143–175 (2001)
Kogan, J.: Means Clustering for Text Data. In: Proceedings of the workshop on Text Mining at the First SIAM International Conference on Data Mining, pp. 54–57 (2001)
Yang, Y., Pedersen, J.O.: A Comparative Study on Feature Selection in Text Categorization. In: Proceedings of 14th International Conference on Machine Learning, pp. 412–420 (1997)
Baker, L.D., McCallum, A.: Distributional Clustering of Words for Text Classification. In: Proceedings of 21st Annual International ACM SIGIR, pp. 96–103 (1998)
Slonim, N., Tishby, N.: The Power of Word Clusters for Text Classification. In: Proceedings of 23rd European Colloquium on Information Retrieval Research (ECIR) (2001)
Bekkerman, R., El-Yaniv, R., Tishby, N., Winter, Y.: Distributional Word Clusters vs. Words for Text Categorization. Journal of Machine Learning Research 1, 1–48 (2002)
Pereira, F., Tishby, N., Lee, L.: Distributional Clustering of English Words. In: 31st Annual Meeting of ACL, pp. 183–190 (1993)
Dhillon, I.S., Mallela, S., Kumar, R.: A Divisive Infromation-Theoretic Feature Clustering Algorithm for Text Classification. Journal of Machine Learning Research 3, 1265–1287 (2003)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Jiang, JY., Chen, JW., Lee, SJ. (2007). A Clustering Scheme for Large High-Dimensional Document Datasets. In: Kang, L., Liu, Y., Zeng, S. (eds) Advances in Computation and Intelligence. ISICA 2007. Lecture Notes in Computer Science, vol 4683. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74581-5_56
Download citation
DOI: https://doi.org/10.1007/978-3-540-74581-5_56
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74580-8
Online ISBN: 978-3-540-74581-5
eBook Packages: Computer ScienceComputer Science (R0)