research-article

Hadoop high availability through metadata replication

Authors:

Feng Wang,

Jie Qiu,

Jie Yang,

Bo Dong,

Xinhui Li,

Ying LiAuthors Info & Claims

CloudDB '09: Proceedings of the first international workshop on Cloud data management

Pages 37 - 44

https://doi.org/10.1145/1651263.1651271

Published: 02 November 2009 Publication History

Get Access

Abstract

Hadoop is widely adopted to support data intensive distributed applications. Many of them are mission critical and require inherent high availability of Hadoop. Unfortunately, Hadoop has no high availability support yet, and it is not trivial to enhance Hadoop. Based on thorough investigation of Hadoop, this paper proposes a metadata replication based solution to enable Hadoop high availability by removing single point of failure in Hadoop. The solution involves three major phases: in initialization phase, each standby/slave node is registered to active/primary node and its initial metadata (such as version file and file system image) are caught up with those of active/primary node; in replication phase, the runtime metadata (such as outstanding operations and lease states) for failover in future are replicated; in failover phase, standby/new elected primary node takes over all communications. The solution presents several unique features for Hadoop, such as runtime configurable synchronization mode. The experiments demonstrate the feasibility and efficiency of our solution.

References

[1]

Apache Hadoop. http://hadoop.apache.org/

Google Scholar

[2]

E. Baldeschwieler and D. Cutting. 2009. State of Hadoop. In Hadoop Summit 2009 (Santa Clara, US, June 10, 2009).

Google Scholar

[3]

A. Stern. 2008. Amazon S3 Down. July 20, 2008. http://www.centernetworks.com/amazon-s3-down-july-2008

Google Scholar

[4]

T. White. 2009. Hadoop: The Definitive Guide. O'Reilly Media, Inc. June 2009.

Digital Library

Google Scholar

[5]

D. Skeen and M. Stonebraker. 1983. A Formal Model of Crash Recovery in a Distributed System. IEEE Transactions on Software Engineering. Vol. 9, Issue 3 (May 1983), 219--228. DOI = http://doi.acm.org/10.1109/TSE.1983.236608

Digital Library

Google Scholar

[6]

M. Burrows. 2006. The Chubby Lock Service for Loosely-Coupled Distributed Systems. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (Seattle, WA, USA, November 06-08, 2006). OSDI'06. USENIX Association, Berkeley, CA, 335--350.

Digital Library

Google Scholar

[7]

Apache Zookeeper. http://hadoop.apache.org/zookeeper/

Google Scholar

[8]

C. Bisciglia. Hadoop HA Configuration. Jul. 22, 2009. http://www.cloudera.com/blog/2009/07/22/hadoop-ha-configuration/

Google Scholar

[9]

E. Sorensen. 2007. Derby: Replication and Availability. MS Thesis. Norwegian University of Science and Technology. June 2007.

Google Scholar

[10]

H. Yadava. The Berkeley DB Book. Apress. Oct. 2007.

Digital Library

Google Scholar

[11]

MySQL. http://www.mysql.com/

Google Scholar

[12]

Torodanhan. 2009. Best Practice: DB2 High Availability Disaster Recovery. Apr. 1, 2009. http://www.ibm.com/developerworks/wikis/display/data/Best+Practice+-+DB2+High+Availability+Disaster+Recovery

Google Scholar

Cited By

View all

Abusalah BQadah TStephen JEugster P(2023)Interminable Flows: A Generic, Joint, Customizable Resiliency Model for Big-Data Streaming PlatformsIEEE Access10.1109/ACCESS.2023.323936511(10762-10776)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3239365
Dhulavvagol PTotad S(2023)Performance Enhancement of Distributed System Using HDFS Federation and ShardingProcedia Computer Science10.1016/j.procs.2023.01.254218:C(2830-2841)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.procs.2023.01.254
Abusalah BSchatzlein DStephen JArdekani MEugster P(2022)Multi-Framework Reliability ApproachIEEE Transactions on Cloud Computing10.1109/TCC.2021.306590610:4(2750-2768)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TCC.2021.3065906
Show More Cited By

Index Terms

Hadoop high availability through metadata replication

Recommendations

G-Hadoop: MapReduce across distributed data centers for data-intensive computing

Recently, the computational requirements for large-scale data-intensive analysis of scientific data have grown significantly. In High Energy Physics (HEP) for example, the Large Hadron Collider (LHC) produced 13 petabytes of data in 2010. This huge ...
Impact of Memory Size on Bigdata Processing based on Hadoop and Spark
RACS '17: Proceedings of the International Conference on Research in Adaptive and Convergent Systems

Hadoop and Spark are well-known big data processing platforms. The main technologies of Hadoop are Hadoop Distributed File System and MapReduce processing. Hadoop stores intermediary data on Hadoop Distributed File System, which is a disk-based ...
Design and Development of a Medical Big Data Processing System Based on Hadoop

Secondary use of medical big data is increasingly popular in healthcare services and clinical research. Understanding the logic behind medical big data demonstrates tendencies in hospital information technology and shows great significance for hospital ...

Comments

Information & Contributors

Information

Published In

CloudDB '09: Proceedings of the first international workshop on Cloud data management

November 2009

62 pages

ISBN:9781605588025

DOI:10.1145/1651263

General Chair:
Xiaofeng Meng
Renmin University of China, China
,
Program Chairs:
Haixun Wang
IBM T. J. Watson Research, USA
,
Ying Chen
IBM China Research Lab, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

hadoop

Qualifiers

Research-article

Conference

CIKM '09

Sponsor:

CIKM '09: Conference on Information and Knowledge Management

November 2, 2009

Hong Kong, China

Acceptance Rates

CloudDB '09 Paper Acceptance Rate 8 of 11 submissions, 73%;

Overall Acceptance Rate 12 of 17 submissions, 71%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

74
Total Citations
View Citations
2,750
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Abusalah BQadah TStephen JEugster P(2023)Interminable Flows: A Generic, Joint, Customizable Resiliency Model for Big-Data Streaming PlatformsIEEE Access10.1109/ACCESS.2023.323936511(10762-10776)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3239365
Dhulavvagol PTotad S(2023)Performance Enhancement of Distributed System Using HDFS Federation and ShardingProcedia Computer Science10.1016/j.procs.2023.01.254218:C(2830-2841)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.procs.2023.01.254
Abusalah BSchatzlein DStephen JArdekani MEugster P(2022)Multi-Framework Reliability ApproachIEEE Transactions on Cloud Computing10.1109/TCC.2021.306590610:4(2750-2768)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TCC.2021.3065906
Qian JKim TJeon M(2021)Reliability of Large Scale GPU Clusters for Deep Learning WorkloadsCompanion Proceedings of the Web Conference 202110.1145/3442442.3452056(179-181)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3442442.3452056
Kim TOh S(2021)Metadata Replication with Synchronous OpCodes Writing for Namenode Multiplexing in Hadoop2021 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS)10.1109/IEMTRONICS52119.2021.9422639(1-7)Online publication date: 21-Apr-2021
https://doi.org/10.1109/IEMTRONICS52119.2021.9422639
Murturi IAvasalcai CTsigkanos CDustdar S(2019)Edge-to-Edge Resource Discovery using Metadata Replication2019 IEEE 3rd International Conference on Fog and Edge Computing (ICFEC)10.1109/CFEC.2019.8733149(1-6)Online publication date: May-2019
https://doi.org/10.1109/CFEC.2019.8733149
Maleki NRahmani AConti M(2019)MapReduce: an infrastructure review and research insightsThe Journal of Supercomputing10.1007/s11227-019-02907-5Online publication date: 8-Jun-2019
https://doi.org/10.1007/s11227-019-02907-5
Nguyen MWon HSon SGil MMoon Y(2019)Prefetching-based metadata management in Advanced Multitenant HadoopThe Journal of Supercomputing10.1007/s11227-017-2019-575:2(533-553)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s11227-017-2019-5
Rajeswari DPrakash MSuresh J(2019)Computational grid scheduling architecture using MapReduce model-based non-dominated sorting genetic algorithmSoft Computing10.1007/s00500-019-03946-zOnline publication date: 27-Mar-2019
https://doi.org/10.1007/s00500-019-03946-z
Chakroborti DRoy BMondal AMostaeen GRoy CSchneider KDeters R(2019)A Data Management Scheme for Micro-Level Modular Computation-Intensive Programs in Big Data PlatformsData Management and Analysis10.1007/978-3-030-32587-9_9(135-153)Online publication date: 21-Dec-2019
https://doi.org/10.1007/978-3-030-32587-9_9
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

G-Hadoop: MapReduce across distributed data centers for data-intensive computing

Impact of Memory Size on Bigdata Processing based on Hadoop and Spark

Design and Development of a Medical Big Data Processing System Based on Hadoop

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tag

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations