skip to main content
10.1145/3318299.3318315acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlcConference Proceedingsconference-collections
research-article

A Distributed PCM Clustering Algorithm Based on Spark

Authors Info & Claims
Published:22 February 2019Publication History

ABSTRACT

With the large-scale growth of data, traditional single-machine data processing methods are difficult to deal with massive data, especially iterative clustering algorithms that require frequent reading and writing operations. On the basis of Spark framework, this paper proposes a distributed possibilistic c-means algorithm based on memory computing, called Spark-PCM. The proposed method improves the related processing of distributed matrix operation and is implemented on the Spark platform. Experimental results show that the proposed Spark-PCM algorithm runs in a linear relationship with the number of nodes and has a good scalability, which indicates that it has higher scalability and adaptability to large-scale data.

References

  1. Krishnapuram, R. and Keller, J. M. 1993. A possibilistic approach to clustering. IEEE Trans. Fuzzy Syst. 1, 2, 98--110. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Krishnapuram, R. and Keller, J. M. 1996. The possibilistic c-means algorithm: insights and recommendations. IEEE Trans. Fuzzy Syst. 4, 3, 385--393. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Bahrampour, S., Moshiri, B., and Salahshoor, K. 2011. Weighted and constrained possibilistic c-means clustering for online fault detection and isolation. Applied Intelligence 35, 2, 269--284. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Zhang, Q., Yang, L. T., Chen, Z., and Li, P. 2017. PPHOPCM: Privacy-preserving high-order possibilistic c-means algorithm for big data clustering with cloud computing. IEEE Trans. Big Data.Google ScholarGoogle ScholarCross RefCross Ref
  5. Dean, J. and Ghemawat, S. 2008. MapReduce: Simplified data processing on large clusters. Communications of the ACM 51, 1, 107--113. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. White, T. 2012. Hadoop, The Definitive Guide. O'Reilly Media, Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Kim, Y. Shim, K., Kim, M. S., and Lee, J. S. 2014. DBCURE-MR: An efficient density-based clustering algorithm for large data using MapReduce. Information Systems 42, 15--35. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I. 2010. Spark: Cluster computing with working sets. In Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (Boston, MA, June 22-25, 2010). HotCloud'10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Leibiusky, J., Eisbruch, G., and Simonassi, D. 2012. Getting Started with Storm. O'Reilly Media, Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Zaharia, M., Chowdhury, M., Das, T., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. 2012. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Shanahan, J. G. and Dai, L. 2015. Large scale distributed data science using apache spark. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2323--2324. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Rathore, M. M., Son, H., Ahmad, A., Paul, A., and jeon, G. 2018. Real-time big data stream processing using GPU with spark over Hadoop ecosystem. International Journal of Parallel Programming 46, 3, 630--646. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Backhoff, O. and Ntoutsi, E. 2017. Scalable online-offline stream clustering in apache spark. In Proceedings of International Conference on Data Mining Workshops, 37--44.Google ScholarGoogle Scholar
  14. Gu, R., Tang, Y., Tian, C., Zhou, H., Li, G., Zheng, X., and Huang, Y. 2017. Improving execution concurrency of large-scale matrix multiplication on distributed data-parallel platforms. IEEE Trans. Parallel Distrib. Syst. 28, 9, 2539--2552.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. A Distributed PCM Clustering Algorithm Based on Spark

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Other conferences
        ICMLC '19: Proceedings of the 2019 11th International Conference on Machine Learning and Computing
        February 2019
        563 pages
        ISBN:9781450366007
        DOI:10.1145/3318299

        Copyright © 2019 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 22 February 2019

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed limited

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader