skip to main content
10.1145/3147213.3147221acmconferencesArticle/Chapter ViewAbstractPublication PagesuccConference Proceedingsconference-collections
research-article

Proactive Re-replication Strategy in HDFS based Cloud Data Center

Published: 05 December 2017 Publication History

Abstract

Cloud storage systems use data replication for fault tolerance, data availability and load balancing. In the presence of node failures, data blocks on the failed nodes are re-replicated to other remaining nodes in the system randomly, thus leading to workload imbalance. Balancing all the server workloads namely, re-replication workload and current running user's application workload during the re-replication phase has not been adequately addressed. With a reactive approach, re-replication can be scheduled based on current resource utilization but by the time replication kicks off, actual resource usage may have changed as resources are continuously in use. In this paper, we propose a proactive re-replication strategy that uses predicted CPU utilization, predicted disk utilization, and popularity of the replicas to perform re-replication effectively while ensuring all the server workloads are balanced. We consider both reliability of a data block and performance status of nodes in making decision for re-replication. Simulation results from synthetic workload data demonstrate that all the servers' utilization is balanced and our approach improves performance in terms of re-replication throughput and re-replication time compared to baseline Hadoop Distributed File System (HDFS). Our proactive approach maintains the balance of resource utilization and avoids the occurrence of servers' overload condition during re-replication.

References

[1]
G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris. 2011. Scarlett: Coping with skewed content popularity in MapReduce clusters. In Proceedings of the Sixth Conference on Computer systems.
[2]
L.A. Barroso and U. Holzle. 2007. The case for energy-propotional computing. Computer 40, 12 (2007).
[3]
L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. 1999. Web caching and zipf-like distributions: evidence and implications. In Proceedings of the 18th Annual Joint Conference of the IEEE Computer and Communications Societies.
[4]
D.-M. Bui, S. Hussain, E.-N. Huh, and S. Lee. 2016. Adaptive replication management in HDFS based on supervised learning. IEEE Transcations on Knowledage and Data Engineering 28, 6 (2016).
[5]
R.N. Calheiros, R. Ranjan, A. Beloglazov, C.A.F.D. Rose, and R. Buyya. 2011. CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software-Practice and Experience 41, 1 (2011).
[6]
R.J. Chansler. 2012. Data availability and durability with the Hadoop distributed file system. The USENIX Magazine 37 (2012).
[7]
A. Cidon, R. Escriva, S. Katti, M. Rosenblum, and E.G. Sirer. 2015. Tiered replication: a cost-effective alternative to full cluster geo-replication. In Proceedings of the 2015 Usenix Annual Technical Conference.
[8]
A. Cidon, S.M. Rumble, R. Stutsman, S. Katti, J. Ousterhout, and M. Rosenblum. 2013. Copysets:reducing the frequency of data loss in cloud storage. In Proceedings of the 2013 USENIX Annual Technical Conference. 37--48.
[9]
W.S. Cleveland. 1979. Robust locally weighted regression and smoothing scatter plots. J. Amer. Statist. Assoc. 74, 368 (1979).
[10]
C. Debians, P.A.-T. Togores, and F. Karakusoglu. 2012. HDFS Replication Simulator. (2012). Retrieved December 1, 2016 from https://github/peteratt/HDFS-Replication-Simulator
[11]
A. Duminuco, E. Biersack, and T. En-Najjary. 2007. Proactive replication in distributed storage systems using machine availability estimation. In Proceedings of the International Conference on Emerging Networking Experiments and Technologies.
[12]
S. Ghemawat, H. Gobioff, and S.-T. Leung. 2003. The Google file system. In Proceedings of the 19th ACM Symposim on Operating Systems Principles.
[13]
N. Grozev and R. Buyya. 2015. Performance modelling and simulation of three-tier applications in cloud and multi-cloud environments. Comput. J. 58, 1 (2015).
[14]
A. Higai, A. Takefusa, H. Nakada, and M. Oguchi. 2014. A study of effective replica reconstruction schemes at node deletion for HDFS. In Proceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.
[15]
A. Higai, A. Takefusa, H. Nakada, and M. Oguchi. 2014. A study of replica reconstruction schemes for multi-rack HDFS clusters. In Proceedings of the IEEE/ACM 7th International Conference on Utility and Cloud Computing.
[16]
G. Kousiouris, G. Vafiadis, and T. Varvarigou. 2013. Enabling proactive data management in virtualized Hadoop clusters based on predicted data activity patterns. In Proceedings of the Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing.
[17]
L.-W. Lee, P. Scheuermann, and R. Vingralek. 2000. File Assignment in parallel I/O systems with minimal variance of service time. IEEE Trans. Comput. 49, 2 (2000).
[18]
D. Magalhaes, R.N. Calheiros, R. Buyya, and D.G.Gomes. 2015. Workload modeling for resource usage analysis and simulation in cloud computing. Journal of Computers and Electrical Engineering 47, C (2015).
[19]
D. Meisner, B.T. Gold, and T.F. Wenisch. 2009. PowerNap: eliminating server idle power. In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems.
[20]
G.A.F. Seber and A.J. Lee. 2003. Linear Regression Analysis (2nd. ed.). John Wiley and Sons.,Inc.
[21]
K. Shvachko, H. Kuang, S. Radia, and R. Chansler. 2010. The Hadoop distributed file system. In Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies.
[22]
T. Shwe and M. Aritsugi. 2016. A re-replication approach in HDFS based cloud data center. In Proceedings of the AUN/SEED-NET Regional Conference on Computer and Information Engineering.
[23]
T. Shwe and M. Aritsugi. {n.d}. A data re-replication scheme and its improvement toward proactive approach. ({n.d}). submitted for publication.
[24]
E. Sit, A. Haeberlen, F. Dabek, B.G. Chun, H. Weatherspoon, R. Morris, M.F. Kaashoek, and J. Kubiatowicz. 2006. Proactive replication for data durability. In Proceedings of the Fifth International Workshop on Peer-to-Peer Systems.
[25]
M. Tang, B.-S. Lee, X. Tang, and C.-K. Yeo. 2006. The impact of data replication on job scheduling performance in the data grid. Future Generation Computer Systems 22, 3 (2006).
[26]
T. White. 2012. Hadoop: The Definitive Guide (3rd. ed.). O'Reilly Media, Inc.
[27]
S. Wu, H. Jiang, and B. Mao. 2015. Proactive data migration for improved storage availability in large-scale data centers. IEEE Trans. Comput. 64, 9 (2015).
[28]
S. Wu, W. Zhu, B. Mao, and K.Li. 2017. PP: Popularity-based Proactive Data Recovery for HDFS RAID Systems. Future Generation Computer Systems (2017).

Cited By

View all
  • (2021)DRF-FTS: A Dynamic Replication Factor Replication Scheme Based on Fault-Tolerant Set2021 IEEE 3rd International Conference on Civil Aviation Safety and Information Technology (ICCASIT)10.1109/ICCASIT53235.2021.9633522(974-979)Online publication date: 20-Oct-2021
  • (2021)Data replication schemes in cloud computing: a surveyCluster Computing10.1007/s10586-021-03283-7Online publication date: 16-Apr-2021
  • (2020)Effective replica management for improving reliability and availability in edge-cloud computing environmentJournal of Parallel and Distributed Computing10.1016/j.jpdc.2020.04.012Online publication date: May-2020
  • Show More Cited By

Index Terms

  1. Proactive Re-replication Strategy in HDFS based Cloud Data Center

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UCC '17: Proceedings of the10th International Conference on Utility and Cloud Computing
    December 2017
    222 pages
    ISBN:9781450351492
    DOI:10.1145/3147213
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 December 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. fault tolerance
    2. hdfs
    3. prediction
    4. replication

    Qualifiers

    • Research-article

    Conference

    UCC '17
    Sponsor:

    Acceptance Rates

    UCC '17 Paper Acceptance Rate 17 of 63 submissions, 27%;
    Overall Acceptance Rate 38 of 125 submissions, 30%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)9
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 22 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)DRF-FTS: A Dynamic Replication Factor Replication Scheme Based on Fault-Tolerant Set2021 IEEE 3rd International Conference on Civil Aviation Safety and Information Technology (ICCASIT)10.1109/ICCASIT53235.2021.9633522(974-979)Online publication date: 20-Oct-2021
    • (2021)Data replication schemes in cloud computing: a surveyCluster Computing10.1007/s10586-021-03283-7Online publication date: 16-Apr-2021
    • (2020)Effective replica management for improving reliability and availability in edge-cloud computing environmentJournal of Parallel and Distributed Computing10.1016/j.jpdc.2020.04.012Online publication date: May-2020
    • (2019)Application of data fragmentation and replication methods in the cloud: a review2019 International Conference on Electronics, Communications and Computers (CONIELECOMP)10.1109/CONIELECOMP.2019.8673249(47-54)Online publication date: Feb-2019
    • (2018)Avoiding Performance Impacts by Re-Replication Workload Shifting in HDFS Based Cloud StorageIEICE Transactions on Information and Systems10.1587/transinf.2018PAP0017E101.D:12(2958-2967)Online publication date: 1-Dec-2018
    • (2018)PRTuner: Proactive-Reactive Re-Replication Tuning in HDFS-based Cloud Data CenterIEEE Cloud Computing10.1109/MCC.2018.0641811205:6(48-57)Online publication date: Nov-2018

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media