ABSTRACT
Distributed data systems systems are used in a variety of settings like online serving, offline analytics, data transport, and search, among other use cases. They let organizations scale out their workloads using cost-effective commodity hardware, while retaining key properties like fault tolerance and scalability. At LinkedIn we have built a number of such systems. A key pattern we observe is that even though they may serve different purposes, they tend to have a lot of common functionality, and tend to use common building blocks in their architectures. One such building block that is just beginning to receive attention is cluster management, which addresses the complexity of handling a dynamic, large-scale system with many servers. Such systems must handle software and hardware failures, setup tasks such as bootstrapping data, and operational issues such as data placement, load balancing, planned upgrades, and cluster expansion.
All of this shared complexity, which we see in all of our systems, motivates us to build a cluster management framework, Helix, to solve these problems once in a general way.
Helix provides an abstraction for a system developer to separate coordination and management tasks from component functional tasks of a distributed system. The developer defines the system behavior via a state model that enumerates the possible states of each component, the transitions between those states, and constraints that govern the system's valid settings. Helix does the heavy lifting of ensuring the system satisfies that state model in the distributed setting, while also meeting the system's goals on load balancing and throttling state changes. We detail several Helix-managed production distributed systems at LinkedIn and how Helix has helped them avoid building custom management components. We describe the Helix design and implementation and present an experimental study that demonstrates its performance and functionality.
- Apache Cassandra. http://cassandra.apache.org.Google Scholar
- Apache Hadoop. http://hadoop.apache.org/.Google Scholar
- Apache Hadoop NextGen MapReduce (YARN). http://hadoop.apache.org/.Google Scholar
- Apache HBase. http://hbase.apache.org/.Google Scholar
- Apache Mesos. http://incubator.apache.org/mesos/.Google Scholar
- Hedwig. https://cwiki.apache.org/ZOOKEEPER/hedwig.html.Google Scholar
- MongoDB. http://www.mongodb.org/.Google Scholar
- SenseiDB. http://www.senseidb.com/.Google Scholar
- Zookeeper. http://zookeeper.apache.org.Google Scholar
- F. Chang et al. Bigtable: A distributed storage system for structured data. In OSDI, 2006. Google ScholarDigital Library
- B. F. Cooper et al. PNUTS: Yahoo!'s hosted data serving platform. In VLDB, 2008. Google ScholarDigital Library
- J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004. Google ScholarDigital Library
- R. Honicky and E. Miller. Replication under scalable hashing: A family of algorithms for scalable decentralized data distribution. In IPDPS, 2004.Google Scholar
- LinkedIn Data Infrastructure Team. Data infrastructure at LinkedIn. In ICDE, 2012.Google Scholar
- J. Shute et al. F1-the fault-tolerant distributed rdbms supporting google's ad business. In SIGMOD, 2012. Google ScholarDigital Library
- M. Zaharia et al. The datacenter needs an operating system. In HotCloud, 2011. Google ScholarDigital Library
Index Terms
- Untangling cluster management with Helix
Recommendations
Multi-nucleation and vectorial folding pathways of large helix protein
Graphical abstractAt present a unified picture of how large real proteins fold is still absent. We simulated the folding of a large eight-helix-bundle protein with a length of 145 amino acids by using a united-residue protein model and observed a ...
Helix Interaction Tool (HIT): a web-based tool for analysis of helix-helix interactions in proteins
Motivation: In many proteins, helix--helix interactions can be critical to establishing protein conformation (folding) and dynamics, as well as determining associations between protein units. However, the determination of a set of rules that guide ...
Predicting helix–helix interactions from residue contacts in membrane proteins
Motivation: Helix–helix interactions play a critical role in the structure assembly, stability and function of membrane proteins. On the molecular level, the interactions are mediated by one or more residue contacts. Although previous studies focused ...
Comments