RDMA-driven MongoDB: An approach of RDMA enhanced NoSQL paradigm for large-Scale data processing

Highlights

• We propose an effective trade-off scheme among various design choices to achieve high-performance RDMA-driven MongoDB.

• We put forward a RDMA context detection algorithm for determining TCP/IP-based or RDMA communication between two RDMA Mongo nodes.

• We design the load-aware buffer registration mechanism for reasonable memory region management of RDMA channels.

• We redesign the oplogs synchronization protocol with RDMA verbs in RDMA_Mongo, including the oplogs initial sync and steady state replication.

• We have implemented RDMA_Mongo based on MongoDB v-4.1.1-59 which achieves better performance than the plain one.

Abstract

With the rapid development of big data and data center networks, NoSQL database has won great popularity for its excellent performance in accelerating the performance of many online and offline big data applications, such as HBase, Cassandra and MongoDB. However, due to massive and frequent Create/Update/Retrieval/Delete (CURD) operations, the traditional TCP/IP protocol stack has difficulty to provide the required request rates and response latency for the large-scale NoSQL system. For example, large-scale data migration or synchronization among multiple clusters in a data center results in competition for network bandwidth with high delay. To mitigate such transmission bottleneck, we propose an approach of RDMA-driven document NoSQL Paradigm’ RDMA_Mongo, based on MongoDB. The performance of CURD operations is enhanced by one-sided Remote Direct Memory Access (RDMA) primitives (such as RDMA Read/Write) without involving the TCP/IP stack or CPU. Evaluation under RDMA-enabled network demonstrates that RDMA_Mongo significantly improves the CURD performance, compared with plain MongoDB. The results show that the average insert throughput increases by approximately 30%, the average delete throughput by over 30%, the update by up to 17% and the query throughput by 15% when facing large-scale data requests.

Introduction

With the rapid increase of large-scale data, which needs to be collected and analyzed online or offline in a data center, the distributed NoSQL (Not Only Structured Query Language) database systems such as MongoDB have been widely adopted in both academia and industry. NoSQL databases are designed not only to cope with the challenges in scalability and agility, which modern applications are confronted with, but also to take advantages of the commodity storage and computing power available today. As a typical implementation of NoSQL databases, MongoDB employs flexible document data model, auto-sharding, replica sets as core features, and is the only NoSQL database to support multi-document transactions currently. CURD document operations, replica sets and auto-sharding in MongoDB all rely on the underlying network communication. Furthermore, networking for big data [50] is extremely vital to accommodate the demand for big data processing. In the case of high concurrency of big data processing, it is found in our evaluation that the transmission load using traditional network protocol stack has become a bottleneck of MongoDB.

Meanwhile, as an emerging underlying network technology in recent years, RDMA (Remote Direct Memory Access), especially RoCE (RDMA over Converged Ethernet), has been frequently adopted in many industrial datacenters [10], [14], [16], [25], [38], [41], [43]. It offers ideal high throughput, low latency and CPU-bypassing for big data applications like HDFS [13], [36], memory file system [22], analysis and modeling on big data [5], [33]. RDMA network protocol uses zero-copy technology and bypass OS kernel to directly access registered memory in remote host.

Fig. 1 shows the difference between TCP and RDMA. In terms of TCP, the data in application buffer is encapsulated by the sockets API in user space. Then the encapsulated data is written to network card through the sockets and TCP/IP layer protocols of OS kernel. However, in terms of RDMA, the data in application buffer is encapsulated by IB Verbs API in user space, and the encapsulated data is then directly written to the RDMA network card (RNIC) without any involvement of the operating system kernel.

The mainstream RDMA technologies include InfiniBand, RoCE and iWARP. RDMA primitives can be divided into message-oriented two-sided verbs (such as RDMA SEND/RECV) and memory-sharing one-sided verbs (such as RDMA READ/WRITE) [26], [27]. The message-passing communication requires that the receiver should start the transmission session and register a communication buffer. After that, the sender prepares to modify the data in the remote receiver’s memory buffer and accomplishes the data exchanges. However, the problem is, that in such a message-oriented mode, data copy between data memory buffer and communication memory buffer is still unavoidable. The RDMA shared-memory mode merges the data memory buffer and communication memory buffer by allowing sender and receiver sides to remain completely passive. Existing RDMA driven data center applications, called RDMA-Enhanced Paradigm, fall into two categories: RDMA-Enhanced System [4], [13], [14], [15], [16], [17], [18], [19], [20], [22], [28], [29], [36], [38], [39], [43], [44], [46], [47], [49] and RDMA-Enhanced Algorithm [2], [6], [35], [42]. For instance, Nessie [4] proposed a fully client-driven key-value store system using RDMA for high performance. APUS [42] proposed the first RDMA-based Paxos consensus algorithm. Existing RDMA-driven systems mainly focus on distributed file system [13], [20], [22], [36], key-value store [4], [14], [15], [16], [28], [38], [43], [44], [49], parallel database [19], relational database [18], [29], as well as memory transaction [17], [46], [47]. However, to the best of our knowledge, there have been no attempts to enhance document-based NoSQL databases in RDMA-enabled networks. Therefore, can we accelerate document-based NoSQL system with RDMA to mitigate the transmission bottleneck?

This paper presents RDMA_Mongo, which is the first RDMA driven document-based NoSQL paradigm. We first demonstrate a detailed analysis of MongoDB network transport designs. Through analyzing the key stages in the NoSQL CURD operations with high concurrency, we find that there are two key challenges: (1) to decrease frequent CPU context switching caused by data transmission in traditional network and (2) to reduce waiting time on the MongoDB client side.

To address the aforementioned challenges, specifically, RD-MA_Mongo automatically detects whether RDMA NICs and related libraries (such as libibverbs, librdmacm) are supported at both local and remote hosts, and then determines whether to use RDMA verbs or traditional network sockets. Second, based on the current system load (especially memory usage), RDMA_Mongo determines the appropriate buffer size on demand and registers RDMA communication memory regions.

Third, RDMA_Mongo introduces non-blocking data transmission with the power of RDMA Completion Queue (CQ). Without modification on the MongoDB existing network transport interface, RDMA_Mongo rewrites the socket implementation exploiting RDMA one-sided shared-memory primitives, which can coexist with traditional network transport mode. Finally, indigenous MongoDB exploits the replica set mechanism to achieve high availability, that is, the MongoDB cluster has one primary node and multiple secondary nodes at the same time. The secondary nodes need to continuously pull oplogs (Operation Log) from the primary node and replay the idempotent oplogs for database synchronization. To accelerate such process, we optimize and redesign the oplogs synchronization protocol based on RDMA primitives.

The main contributions of our work can be summarized as follows.

• We analyze thoroughly all the various design choices of RDMA-driven paradigm and demonstrate an effective trade-off among different design options for high performance RDMA_Mongo.

• We put forward a RDMA context detection algorithm for determining TCP/IP-based or RDMA communication between two RDMA_Mongo nodes and also propose load-aware buffer registration mechanism for reasonable memory region management of RDMA channels.

• We redesign the oplogs synchronization protocol with RDMA primitives in RDMA_Mongo, including the optimization of oplogs initial sync and steady state replication, in which RDMA completion event channel is used to receive messages asynchronously.

• We have implemented RDMA_Mongo based on MongoDB version 4.1.1-59. The detailed evaluation in RDMA-enabled network demonstrates that RDMA_Mongo significantly improves the CURD performances in average insert by approximately 30%, update by 17%, query by 15% and delete throughput by over 30%, when facing large-scale data.

The following section, that is Section 2, describes the typical NoSQL categories, RDMA and recent efforts on RDMA-Enhanced Paradigm, then introduces the problem statement. Section 3 demonstrates the overview of RDMA_Mongo architecture and discusses the tradeoff among different design choices. In Section 4, RDMA context detection algorithm and load-aware buffer registration mechanism are proposed, and the detail of RDMA-enabled oplogs synchronization is given. In Section 5, we evaluate our system and compare it against the plain MongoDB. In Section 6, we make extensive investigations of recent works related to modern network hardware, kernel-bypass networking, RDMA protocol stack optimization, and RDMA-driven NoSQL. Section 7 comes to the conclusion of this paper and our expectation for future work.