skip to main content
research-article

DxHash: A Memory-saving Consistent Hashing Algorithm

Published: 22 February 2024 Publication History

Abstract

Consistent Hashing (CH) algorithms are widely adopted in networks and distributed systems for their ability to achieve load balancing and minimize disruptions. However, the rise of the Internet of Things (IoT) has introduced new challenges for existing CH algorithms, characterized by high memory usage and update overhead. This article presents DxHash, a novel CH algorithm based on repeated pseudo-random number generation. DxHash offers significant benefits, including a remarkably low memory footprint, high lookup throughput, and minimal update overhead. Additionally, we introduce a weighted variant of DxHash, enabling adaptable weight adjustments to handle heterogeneous load distribution. Through extensive evaluation, we demonstrate that DxHash outperforms AnchorHash, a state-of-the-art CH algorithm, in terms of the reduction of up to 98.4% in memory footprint and comparable performance in lookup and update.

References

[1]
Shahbaz Akhtar Abid, Mazliza Othman, and Nadir Shah. 2014. A survey on DHT-based routing for large-scale mobile ad hoc networks. ACM Comput. Surv. 47, 2, Article 20 (Aug.2014), 46 pages. DOI:
[2]
Stephanos Androutsellis-Theotokis and Diomidis Spinellis. 2004. A survey of peer-to-peer content distribution technologies. ACM Computing Surveys (CSUR) 36, 4 (2004), 335–371.
[3]
Ben Appleton and Michael O’Reilly. 2015. Multi-probe consistent hashing. arXiv preprint arXiv:1505.00062 (2015).
[4]
Elaine B. Barker, William C. Barker, William E. Burr, W. Timothy Polk, and Miles E. Smid. 2007. SP 800-57. Recommendation for Key Management, Part 1: General (Revised), Technical Report, National Institute of Standards & Technology, Gaithersburg, MD.
[5]
Chanwoo Chung, Jinhyung Koo, Junsu Im, Arvind, and Sungjin Lee. 2019. LightStore: Software-defined network-attached key-value drives. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’19). Association for Computing Machinery, New York, NY, 939–953. DOI:
[7]
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: Amazon’s highly available key-value store. ACM SIGOPS Operating Systems Review 41, 6 (2007), 205–220.
[8]
Daniel E. Eisenbud, Cheng Yi, Carlo Contavalli, Cody Smith, Roman Kononov, Eric Mann-Hielscher, Ardas Cilingiroglu, Bin Cheyney, Wentao Shang, and Jinnah Dylan Hosein. 2016. Maglev: A fast and reliable software network load balancer. In 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI’16). USENIX Association, Santa Clara, CA, 523–535. https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud
[9]
Xiang Fu, Can Peng, and Weihong Han. 2015. A consistent hashing based data redistribution algorithm. In Intelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques, Xiaofei He, Xinbo Gao, Yanning Zhang, Zhi-Hua Zhou, Zhi-Yong Liu, Baochuan Fu, Fuyuan Hu, and Zhancheng Zhang (Eds.). Springer International Publishing, Cham, 559–566.
[10]
Pulkit Goel, Kumar Rishabh, and Vasudeva Varma. 2017. An alternate load distribution scheme in DHTs. In 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom’17). 218–222. DOI:
[11]
Harshit Gupta, Zhuangdi Xu, and Umakishore Ramachandran. 2018. DataFog: Towards a holistic data management platform for the IoT age at the network edge. In USENIX Workshop on Hot Topics in Edge Computing (HotEdge’18). USENIX Association, Boston, MA. https://www.usenix.org/conference/hotedge18/presentation/gupta
[12]
David Karger, Eric Lehman, Tom Leighton, Rina Panigrahy, Matthew Levine, and Daniel Lewin. 1997. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web. In Proceedings of the 29th Annual ACM Symposium on Theory of Computing (STOC’97). Association for Computing Machinery, New York, NY, 654–663. DOI:
[13]
David Karger, Alex Sherman, Andy Berkheimer, Bill Bogstad, Rizwan Dhanidina, Ken Iwamoto, Brian Kim, Luke Matkins, and Yoav Yerushalmi. 1999. Web caching with consistent hashing. Computer Networks 31, 11–16 (1999), 1203–1213.
[14]
Shaishav Kumar and Raghavendra Udupa. 2011. Learning hash functions for cross-view similarity search. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Volume 2 (IJCAI’11). AAAI Press, 1360–1365.
[15]
Avinash Lakshman and Prashant Malik. 2010. Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review 44, 2 (2010), 35–40.
[16]
John Lamping and Eric Veach. 2014. A Fast, Minimal Memory, Consistent Hash Algorithm. (2014). arxiv:cs.DS/1406.2294
[17]
Zaoxing Liu, Zhihao Bai, Zhenming Liu, Xiaozhou Li, Changhoon Kim, Vladimir Braverman, Xin Jin, and Ion Stoica. 2019. DistCache: Provable load balancing for large-scale storage systems with distributed caching. In 17th USENIX Conference on File and Storage Technologies (FAST’19). USENIX Association, Boston, MA, 143–157. https://www.usenix.org/conference/fast19/presentation/liu
[18]
Petar Maymounkov and David Mazieres. 2002. Kademlia: A peer-to-peer information system based on the xor metric. In International Workshop on Peer-to-peer Systems. Springer, 53–65.
[19]
Gal Mendelson, Shay Vargaftik, Katherine Barabash, Dean H. Lorenz, Isaac Keslassy, and Ariel Orda. 2021. AnchorHash: A scalable consistent hash. IEEE/ACM Transactions on Networking 29, 2 (2021), 517–528. DOI:
[20]
Satoshi Nakamoto. 2008. Bitcoin: A peer-to-peer electronic cash system. https://bitcoin.org/bitcoin.pdf
[21]
Yuichi Nakatani. 2021. Structured allocation-based consistent hashing with improved balancing for cloud infrastructure. IEEE Transactions on Parallel and Distributed Systems 32, 9 (2021), 2248–2261. DOI:
[22]
Andreas N. Philippou, Costas Georghiou, and George N. Philippou. 1983. A generalized geometric distribution and some of its properties. Statistics & Probability Letters 1, 4 (1983), 171–175.
[23]
Jiwu Shu, Youmin Chen, Qing Wang, Bohong Zhu, Junru Li, and Youyou Lu. 2020. TH-DPMS: Design and implementation of an RDMA-enabled distributed persistent memory storage system. ACM Transactions on Storage 16, 4, Article 24 (Oct.2020), 31 pages. DOI:
[24]
Ronak Singhal. 2008. Inside Intel® core microarchitecture (nehalem). In 2008 IEEE Hot Chips 20 Symposium (HCS’08). IEEE, 1–25.
[25]
I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek, and H. Balakrishnan. 2003. Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Transactions on Networking 11, 1 (2003), 17–32. DOI:
[26]
D. G. Thaler and C. V. Ravishankar. 1998. Using name-based mappings to increase hit rates. IEEE/ACM Transactions on Networking 6, 1 (1998), 1–14. DOI:
[27]
Xiaoming Wang and Dmitri Loguinov. 2007. Load-balancing performance of consistent hashing: Asymptotic analysis of random node join. IEEE/ACM Transactions on Networking 15, 4 (2007), 892–905. DOI:
[28]
Yair Weiss, Antonio Torralba, and Rob Fergus. 2009. Spectral hashing. In Proceedings of the 2009 Conference on Computer Vision and Pattern Recognition, 293–300.
[29]
Chenggang Wu, Vikram Sreekanti, and Joseph M. Hellerstein. 2019. Autoscaling tiered cloud storage in anna. Proceedings of the VLDB Endowment 12, 6 (Feb.2019), 624–638. DOI:
[30]
Di Wu, Dmitri I. Arkhipov, Eskindir Asmare, Zhijing Qin, and Julie A. McCann. 2015. UbiFlow: Mobility management in urban-scale software defined IoT. In 2015 IEEE Conference on Computer Communications (INFOCOM’15). 208–216. DOI:

Cited By

View all
  • (2025)Cryptocurrency and Future of MoneyAlgorithmic Training, Future Markets, and Big Data for Finance Digitalization10.4018/979-8-3693-6386-7.ch010(219-260)Online publication date: 24-Jan-2025
  • (2025)Constructing a scalable key-value store engine on multidisk systemThe Journal of Supercomputing10.1007/s11227-024-06725-281:1Online publication date: 1-Jan-2025
  • (2024)On the Modification of Ring Consistent Hash Uniformity: A Case Study2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC54092.2024.10831430(4778-4783)Online publication date: 6-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet Technology
ACM Transactions on Internet Technology  Volume 24, Issue 1
February 2024
178 pages
EISSN:1557-6051
DOI:10.1145/3613514
  • Editor:
  • Ling Liu
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 February 2024
Online AM: 03 November 2023
Accepted: 27 October 2023
Revised: 21 September 2023
Received: 09 March 2023
Published in TOIT Volume 24, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. DxHash
  2. consistent hashing
  3. load balance
  4. memory saving

Qualifiers

  • Research-article

Funding Sources

  • National Natural Science Foundation of China (NSFC)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)347
  • Downloads (Last 6 weeks)9
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Cryptocurrency and Future of MoneyAlgorithmic Training, Future Markets, and Big Data for Finance Digitalization10.4018/979-8-3693-6386-7.ch010(219-260)Online publication date: 24-Jan-2025
  • (2025)Constructing a scalable key-value store engine on multidisk systemThe Journal of Supercomputing10.1007/s11227-024-06725-281:1Online publication date: 1-Jan-2025
  • (2024)On the Modification of Ring Consistent Hash Uniformity: A Case Study2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC54092.2024.10831430(4778-4783)Online publication date: 6-Oct-2024
  • (2024) JumpBackHash : Say Goodbye to the Modulo Operation to Distribute Keys Uniformly to Buckets Software: Practice and Experience10.1002/spe.338555:3(559-570)Online publication date: 5-Nov-2024

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media