tutorial

DaRPC: Data Center RPC

Authors:

Patrick Stuedi,

Animesh Trivedi,

Bernard Metzler,

Jonas PfefferleAuthors Info & Claims

SOCC '14: Proceedings of the ACM Symposium on Cloud Computing

Pages 1 - 13

https://doi.org/10.1145/2670979.2670994

Published: 03 November 2014 Publication History

Abstract

Remote Procedure Call (RPC) has been the cornerstone of distributed systems since the early 80s. Recently, new classes of large-scale distributed systems running in data centers are posing extra challenges for RPC systems in terms of scaling and latency. We find that existing RPC systems make very poor usage of resources (CPU, memory, network) and are not ready to handle these upcoming workloads.

In this paper we present DaRPC, an RPC framework which uses RDMA to implement a tight integration between RPC message processing and network processing in user space. DaRPC efficiently distributes computation, network resources and RPC resources across cores and memory to achieve a high aggregate throughput (2-3M ops/sec) at a very low per-request latency (10μs with iWARP). In the evaluation we show that DaRPC can boost the RPC performance of existing distributed systems in the cloud by more than an order of magnitude for both throughput and latency.

References

[1]

Perf: Linux Profiling with Performance Counters. http://perf.wiki.kernel.org/.

[2]

Zookeeper Performance. http://wiki.apache.org/hadoop/ZooKeeper/.

[3]

M. Balakrishnan, D. Malkhi, T. Wobber, M. Wu, V. Prabhakaran, M. Wei, J. D. Davis, S. Rao, T. Zou, and A. Zuck. Tango: Distributed Data Structures over a Shared Log. In Proceedings of SOSP'13, pages 325--340. ISBN 978-1-4503-2388-8. URL http://doi.acm.org/10.1145/2517349.2522732.

Digital Library

[4]

A. Bilas and E. W. Felten. Fast RPC on the SHRIMP Virtual Memory Mapped Network Interface. J. Parallel Distrib. Comput., 40(1):138--146, Jan. 1997. ISSN 0743-7315. URL http://dx.doi.org/10.1006/jpdc.1996.1272.

Digital Library

[5]

A. D. Birrell and B. J. Nelson. Implementing Remote Procedure Calls. ACM Trans. Comput. Syst., 2(1):39--59, Feb. 1984. ISSN 0734-2071. URL http://doi.acm.org/10.1145/2080.357392.

Digital Library

[6]

M. Dobrescu, N. Egi, K. Argyraki, B.-G. Chun, K. Fall, G. Iannaccone, A. Knies, M. Manesh, and S. Ratnasamy. RouteBricks: Exploiting Parallelism to Scale Software Routers. In Proceedings of the ACM SIGOPS 22Nd Symposium on Operating Systems Principles, SOSP '09, pages 15--28, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-752-3. URL http://doi.acm.org/10.1145/1629575.1629578.

Digital Library

[7]

A. Dragojević, D. Narayanan, O. Hodson, and M. Castro. FaRM: Fast Remote Memory. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI'14, pages 401--414, Berkeley, CA, USA, 2014. USENIX Association. ISBN 978-1-931971-09-6. URL http://dl.acm.org/citation.cfm?id=2616448.2616486.

Digital Library

[8]

S. Han, S. Marshall, B.-G. Chun, and S. Ratnasamy. MegaPipe: A New Programming Interface for Scalable Network I/O. In Proceedings of OSDI'12, pages 135--148. ISBN 978-1-931971-96-6. URL http://dl.acm.org/citation.cfm?id=2387880.2387894.

Digital Library

[9]

E. Y. Jeong, S. Woo, M. Jamshed, H. Jeong, S. Ihm, D. Han, and K. Park. mTCP: A Highly Scalable Userlevel TCP Stack for Multicore Systems. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI'14, pages 489--502, Berkeley, CA, USA, 2014. USENIX Association. ISBN 978-1-931971-09-6. URL http://dl.acm.org/citation.cfm?id=2616448.2616493.

Digital Library

[10]

D. B. Johnson and W. Zwaenepoel. The Peregrine High-performance RPC System. Softw. Pract. Exper., 23(2):201--221, Feb. 1993. ISSN 0038-0644. URL http://dx.doi.org/10.1002/spe.4380230205.

Digital Library

[11]

A. Kalia, M. Kaminsky, and D. G. Andersen. Using RDMA Efficiently for Key-value Services. In Proceedings of the 2014 ACM Conference on SIGCOMM, SIGCOMM '14, pages 295--306, New York, NY, USA, 2014. ACM. ISBN 978-1-4503-2836-4. URL http://doi.acm.org/10.1145/2619239.2626299.

Digital Library

[12]

H. Lim, D. Han, D. G. Andersen, and M. Kaminsky. MICA: A Holistic Approach to Fast In-memory Key-value Storage. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI'14, pages 429--444, Berkeley, CA, USA, 2014. USENIX Association. ISBN 978-1-931971-09-6. URL http://dl.acm.org/citation.cfm?id=2616448.2616488.

Digital Library

[13]

X. Lu, N. S. Islam, M. Wasi-Ur-Rahman, J. Jose, H. Subramoni, H. Wang, and D. K. Panda. High-Performance Design of Hadoop RPC with RDMA over InfiniBand. In Proceedings of the 2013 42Nd International Conference on Parallel Processing, ICPP '13, pages 641--650, Washington, DC, USA, 2013. IEEE Computer Society. ISBN 978-0-7695-5117-3. URL http://dx.doi.org/10.1109/ICPP.2013.78.

Digital Library

[14]

Y. Mao, E. Kohler, and R. T. Morris. Cache Craftiness for Fast Multicore Key-value Storage. In Proceedings of Eurosys'12, pages 183--196. ISBN 978-1-4503-1223-3. URL http://doi.acm.org/10.1145/2168836.2168855.

Digital Library

[15]

C. Mitchell, Y. Geng, and J. Li. Using One-sided RDMA Reads to Build a Fast, CPU-efficient Key-value Store. In Proceedings of the 2013 USENIX Conference on Annual Technical Conference, USENIX ATC'13, pages 103--114, Berkeley, CA, USA, 2013. USENIX Association. URL http://dl.acm.org/citation.cfm?id=2535461.2535475.

Digital Library

[16]

D. Ongaro, S. M. Rumble, R. Stutsman, J. Ousterhout, and M. Rosenblum. Fast Crash Recovery in RAMCloud. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP '11, pages 29--41, New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0977-6. URL http://doi.acm.org/10.1145/2043556.2043560.

Digital Library

[17]

D. Pariag, T. Brecht, A. Harji, P. Buhr, A. Shukla, and D. R. Cheriton. Comparing the Performance of Web Server Architectures. In Proceedings of Eurosys'07, pages 231--243. ISBN 978-1-59593-636-3. URL http://doi.acm.org/10.1145/1272996.1273021.

Digital Library

[18]

D. Peng and F. Dabek. Large-scale Incremental Processing Using Distributed Transactions and Notifications. In Proceedings of OSDI'10, pages 1--15. URL http://dl.acm.org/citation.cfm?id=1924943.1924961.

Digital Library

[19]

A. Pesterev, J. Strauss, N. Zeldovich, and R. T. Morris. Improving Network Connection Locality on Multicore Systems. In Proceedings of Eurosys'12, pages 337--350. ISBN 978-1-4503-1223-3. URL http://doi.acm.org/10.1145/2168836.2168870.

Digital Library

[20]

H. V. Shah, C. Pu, and R. S. Madukkarumukumana. High Performance Sockets and RPC over Virtual Interface (VI) Architecture. In Proceedings of the Third International Workshop on Network-Based Parallel Computing: Communication, Architecture, and Applications, CANPC '99, pages 91--107, London, UK, UK, 1999. Springer-Verlag. ISBN 3-540-65915-3. URL http://dl.acm.org/citation.cfm?id=646093.680566.

Digital Library

[21]

L. Shalev, J. Satran, E. Borovik, and M. Ben-Yehuda. IsoStack: Highly Efficient Network Processing on Dedicated Cores. In Proceedings of USENIX ATC'10, pages 5--5. URL http://dl.acm.org/citation.cfm?id=1855840.1855845.

Digital Library

[22]

K. V. Shvachko. HDFS Scalability: The Limits to Growth. 2010.

[23]

P. Stuedi, B. Metzler, and A. Trivedi. jVerbs: Ultra-low Latency for Data Center Applications. In Proceedings of SOCC'13, pages 10:1--10:14. ISBN 978-1-4503-2428-1. URL http://doi.acm.org/10.1145/2523616.2523631.

Digital Library

[24]

C. A. Thekkath and H. M. Levy. Limits to Low-latency Communication on High-speed Networks. ACM Trans. Comput. Syst., 11(2):179--203, May 1993. ISSN 0734-2071. URL http://doi.acm.org/10.1145/151244.151247.

Digital Library

[25]

A. Trivedi, B. Metzler, and P. Stuedi. A Case for RDMA in Clouds: Turning Supercomputer Networking into Commodity. In Proceedings of the 2nd APSys'11, pages 17:1--17:5. ISBN 978-1-4503-1179-3. URL http://doi.acm.org/10.1145/2103799.2103820.

Digital Library

[26]

V. Vasudevan, M. Kaminsky, and D. G. Andersen. Using Vector Interfaces to Deliver Millions of IOPS from a Networked Key-value Storage Server. In Proceedings of SOCC'12, pages 8:1--8:13. ISBN 978-1-4503-1761-0. URL http://doi.acm.org/10.1145/2391229.2391237.

Digital Library

[27]

M. Welsh, D. Culler, and E. Brewer. SEDA: An Architecture for Well-conditioned, Scalable Internet Services. In Proceedings of SOSP'01, pages 230--243. ISBN 1-58113-389-8. URL http://doi.acm.org/10.1145/502034.502057.

Digital Library

Cited By

Agarwal SCai QAgarwal RShmoys DVahdat AVanbever LZhang I(2024)HarmonyProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691845(329-343)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691845
Nguyen TBecchi M(2024)A Transducers-based Programming Framework for Efficient Data TransformationProceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques10.1145/3656019.3676891(66-77)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3656019.3676891
Luo LZhao GXu HChung CXie L(2024)SMART: Dual-channel Southbound Message Delivery in Clouds with Rate Estimation2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS)10.1109/IWQoS61813.2024.10682836(1-10)Online publication date: 19-Jun-2024
https://doi.org/10.1109/IWQoS61813.2024.10682836
Show More Cited By

Index Terms

DaRPC: Data Center RPC
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Communications management

Recommendations

Coyote: a system for constructing fine-grain configurable communication services

Communication-oriented abstractions such as atomic multicast, group RPC, and protocols for location-independent mobile computing can simplify the development of complex applications built on distributed systems. This article describes Coyote, a system ...
MOM vs. RPC: Communication Models for Distributed Applications

An important aspect of designing a distributed application is the communication model that connects its various components. Some middleware solutions offer two of the most important communication paradigms: asynchronous messaging and remote procedure ...
Designing an Agent Synthesis System for Cross-RPC Communication

Remote procedure call (RPC) is the most popular paradigm used today to build distributed systems and applications. As a consequence, the term "RPC" has grown to include a range of vastly different protocols above the transport layer. A resulting problem ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SOCC '14: Proceedings of the ACM Symposium on Cloud Computing

November 2014

383 pages

ISBN:9781450332521

DOI:10.1145/2670979

Conference Chairs:
Ed Lazowska,
Doug Terry,
Program Chairs:
Remzi H. Arpaci-Dusseau,
Johannes Gehrke

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Tutorial
Research
Refereed limited

Conference

SOCC '14

Sponsor:

SOCC '14: ACM Symposium on Cloud Computing

November 3 - 5, 2014

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
1,178
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)3

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Agarwal SCai QAgarwal RShmoys DVahdat AVanbever LZhang I(2024)HarmonyProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691845(329-343)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691845
Nguyen TBecchi M(2024)A Transducers-based Programming Framework for Efficient Data TransformationProceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques10.1145/3656019.3676891(66-77)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3656019.3676891
Luo LZhao GXu HChung CXie L(2024)SMART: Dual-channel Southbound Message Delivery in Clouds with Rate Estimation2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS)10.1109/IWQoS61813.2024.10682836(1-10)Online publication date: 19-Jun-2024
https://doi.org/10.1109/IWQoS61813.2024.10682836
Sun HTan YWu YZhu JHuang QYao XZhang G(2024) RB 2 : Narrow the Gap between RDMA Abstraction and Performance via a Middle Layer IEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621169(1071-1080)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621169
Zhang JChen XZhang YWang Z(2024)DmRPC: Disaggregated Memory-aware Datacenter RPC for Data-intensive Applications2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00291(3796-3809)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00291
An BSung H(2023)Efficient I/O Merging Scheme for Distributed File SystemsSymmetry10.3390/sym1502042315:2(423)Online publication date: 5-Feb-2023
https://doi.org/10.3390/sym15020423
Gao KSun CWang SLi DZhou YLiu HZhu LZhang MDeng XZhou CLu L(2023)Buffer-Based High-Coverage and Low-Overhead Request Event Monitoring in the CloudIEEE/ACM Transactions on Networking10.1109/TNET.2022.322461031:4(1732-1747)Online publication date: Aug-2023
https://doi.org/10.1109/TNET.2022.3224610
Zhang TZhou HHuang CTian CZhang WWang XWang YAbdelmoniem ATan MDou WChen G(2023)Achieving Zero-copy Serialization for Datacenter RPC2023 IEEE International Performance, Computing, and Communications Conference (IPCCC)10.1109/IPCCC59175.2023.10253859(304-312)Online publication date: 17-Nov-2023
https://doi.org/10.1109/IPCCC59175.2023.10253859
Yu YZhao GXu HTu HLuo LXie L(2023)SNIP: Southbound Message Delivery with In-network Pruning in Clouds2022 IEEE 28th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS56603.2022.00085(609-616)Online publication date: Jan-2023
https://doi.org/10.1109/ICPADS56603.2022.00085
Taranov KByan SMarathe VHoefler TIves ZBonifati AEl Abbadi A(2022)KafkaDirect: Zero-copy Data Access for Apache Kafka over RDMA NetworksProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3526056(2191-2204)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3526056
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten