Abstract
MapReduce is a popular model in which the dataflow takes the form of a directed acyclic graph of operators. But it lacks built-in support for iterative programs, which arise naturally in many clustering applications. Based on micro-cluster and equivalence relation, we design a clustering algorithm which can be easily parallelized in MapReduce and done in quite a few MapReduce rounds. Experiments show that our algorithm not only runs fast and obtains good accuracy but also scales well and possesses high speedup.
The work described in this paper is supported by Foundation of Guangxi Key Laboratory of Trustworthy Software, China (kx201116) and Educational Commission of Guangxi Province, China(201204LX122).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Apache Hadoop, http://hadoop.apache.org
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of Operating Systems Design and Implementation, San Francisco, CA, pp. 137–150 (2004)
Lloyd, S.: Least squares quantization in PCM. IEEE Transactions on Information Theory 28(2), 129–136 (1982)
Aggarwal, C., Han, J., Wang, J., Yu, P.: A Framework for Clustering Evolving Data Streams. In: Proceedings of the 29th International Conference on Very Large Data Bases, Berlin, German, pp. 81–92 (2003)
Zhang, T., Ramakrishnan, R., Livny, M.: BIRCH: an efficient data clustering method for very large databases. In: Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, Montreal, Quebec, Canada, pp. 103–114 (1996)
Amini, A., Wah, T.: Density micro-clustering algorithms on data streams: A review. In: International Conference on Data Mining and Applications (ICDMA), Hong Kong, China, pp. 410–414 (2011)
Papadimitriou, S., Sun, J.: DisCo: Distributed Co-clustering with Map-Reduce: A Case Study Towards Petabyte-Scale End-to-End Mining. In: Proceedings of the 8th IEEE International Conference on Data Mining, Pisa, Italy, pp. 512–521 (2008)
Ene, A., Im, S., Moseley, B.: Fast clustering using MapReduce. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, California, USA, pp. 681–689 (2011)
Bahmani, B., Moseley, B., Vattani, A., Kumar, R., Vassilvitskii, S.: Scalable k-means++. Proceedings of the VLDB Endowment 5(7), 622–633 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Miao, Y., Zhang, J., Feng, H., Qiu, L., Wen, Y. (2013). A Fast Algorithm for Clustering with MapReduce. In: Guo, C., Hou, ZG., Zeng, Z. (eds) Advances in Neural Networks – ISNN 2013. ISNN 2013. Lecture Notes in Computer Science, vol 7951. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39065-4_64
Download citation
DOI: https://doi.org/10.1007/978-3-642-39065-4_64
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39064-7
Online ISBN: 978-3-642-39065-4
eBook Packages: Computer ScienceComputer Science (R0)