Abstract
Short-range queries frequently occur in real-world scenarios. Traditional LSM-Tree-based key-value storage systems handle range query requests using an iterator mechanism. However, the data required for short-range queries typically comes from only a few iterators, and the construction of unnecessary iterators in the iterator mechanism degrades range query performance. To address this issue, we propose a MemTable Retention mechanism for Level 0 and an Iterator Reduction mechanism for other levels in LSM-Tree. The former retains the Immutable MemTable, which should have been persisted to Level 0, in memory to reduce the read overhead from Level 0 during short-range queries. The latter reduces the overhead of creating unnecessary iterators through the Level Filter Table. We developed a prototype system, RIOKV, based on LevelDB. Experimental results show that, under workloads with intensive short-range queries, RIOKV’s throughput is approximately 1.18\(-\)1.58 times that of LevelDB, 1.01\(-\)1.24 times that of RemixDB, and 1.06\(-\)1.46 times that of RocksDB. In workloads with mixed range queries and writes, RIOKV’s throughput is about 1.11\(-\)3.50 times that of LevelDB, 0.94\(-\)1.16 times that of RemixDB, and 1.28\(-\)3.24 times that of RocksDB.
Similar content being viewed by others
References
Braun Lucas, Etter Thomas, Gasparis Georgios, Kaufmann Martin, Kossmann Donald, Widmer Daniel, Avitzur Aharon, Iliopoulos Anthony, Levy Eliezer, Liang Ning (2015) Proceedings of the 2015 Analytics in motion: High performance event-processing and real-time analytics in the same database. In: ACM SIGMOD International Conference on Management of Data 251–264
Chen Guoqiang Jerry, Wiener Janet L, Iyer Shridhar, Jaiswal Anshul, Lei Ran, Simha Nikhil, Wang Wei, Wilfong Kevin, Williamson Tim, Yilmaz Serhat (2016) Realtime data processing at facebook. In: Proceedings of the 2016 In International Conference on Management of Data 1087–1098
Goel Anil K, Pound Jeffrey, Auch Nathan, Bumbulis Peter, MacLean Scott, Färber Franz, Gropengiesser Francis, Mathis Christian, Bodner Thomas, Lehner Wolfgang (2015) Towards scalable real-time analytics: An architecture for scale-out of olxp workloads. Proceedings of the VLDB Endowment 8(12):1716–1727
Yang Juncheng, Yue Yao, Rashmi KV (2021) A large-scale analysis of hundreds of in-memory key-value cache clusters at twitter. ACM Trans Storage (TOS) 17(3):1–35
DeCandia Giuseppe, Hastorun Deniz, Jampani Madan, Kakulapati Gunavardhan, Lakshman Avinash, Pilchin Alex, Sivasubramanian Swaminathan, Vosshall Peter, Vogels Werner (2007) Dynamo: Amazon’s highly available key-value store. ACM SIGOPS Operating Syst Rev 41(6):205–220
Debnath Biplob, Sengupta Sudipta, Li Jin (2011) Skimpystash: Ram space skimpy key-value store on flash-based storage. Proceedings of the 2011 ACM In SIGMOD International Conference on Management of data 56:25–36
Dinh Tien Tuan Anh, Wang Ji, Chen Gang, Liu Rui, Ooi Beng Chin, Tan Kian-Lee (2017) Blockbench: A framework for analyzing private blockchains. Proceedings of the 2017 In ACM international conference on management of data 56:1085–1100
O’Neil Patrick, Cheng Edward, Gawlick Dieter, O’Neil Elizabeth (1996) The log-structured merge-tree (lsm-tree). Acta Informatica 33:351–385
Chang Fay, Dean Jeffrey, Ghemawat Sanjay, Hsieh Wilson C, Wallach Deborah A, Burrows Mike, Chandra Tushar, Fikes Andrew, Gruber Robert E (2008) Bigtable: A distributed storage system for structured data. ACM Trans Comput Syst (TOCS) 26(2):1–26
LevelDB. 2021 https://github.com/google/leveldb,
RocksDB. 2022 http://rocksdb.org/,
Cassandra. http://cassandra.apache.org/, 2020
HBase. 2020 http://hbase.apache.org/,
Vora Mehul Nalin (2011) Hadoop-hbase for large-scale data. In: Proceedings of 2011 International Conference on Computer Science and Network Technology, volume 1, pages 601–605. IEEE
Influx DB: (2022) Purpose-Built Open Source Time Series Database. https://www.influxdata.com/
Siying Dong, Andrew Kryczka, Yanqin Jin, Michael Stumm (2021) Evolution of development priorities in key-value stores serving large-scale applications: The rocksdb experience. In: 19th USENIX Conference on File and Storage Technologies (FAST 21), pages 33–49,
Berk Atikoglu, Yuehai Xu, Eitan Frachtenberg, Song Jiang, Mike Paleczny (2012) Workload analysis of a large-scale key-value store. In: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International conference on Measurement and Modeling of Computer Systems, 53–64,
Cooper, Brian F, Silberstein Adam, Tam Erwin, Ramakrishnan Raghu, Sears Russell (2010)Benchmarking cloud serving systems with ycsb. In: Proceedings of the 1st ACM Symposium on Cloud Computing, 143–154
Cao Zhichao, Dong Siying, Vemuri Sagar, Du David HC (2020) Characterizing, modeling, and benchmarking rocksdb key-value workloads at facebook. In: 18th USENIX Conference on File and Storage Technologies (FAST 20), 209–223
Wang Ziwei, Zhong Zheng, Guo Jiarui, Wu Yuhan, Li Haoyu, Yang Tong, Tu Yaofeng, Zhang Huanchen, Cui Bin (2023) Rencoder: A space-time efficient range filter with local encoder. In: 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 2036–2049. IEEE
Luo Siqiang, Chatterjee Subarna, Ketsetsidis Rafael, Dayan Niv, Qin Wilson, Idreos Stratos (2020) Rosetta: A robust space-time optimized range filter for key-value stores. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pages 2071–2086,
Vaidya Kapil, Chatterjee Subarna, Knorr Eric, Mitzenmacher Michael, Idreos Stratos, Kraska Tim (2022) Snarf: a learning-enhanced range filter. Proceed VLDB Endowment 15(8):1632–1644
Knorr, Eric R, Lemaire Baptiste, Lim Andrew, Luo Siqiang, Zhang Huanchen, Idreos Stratos, Mitzenmacher Michael (2022) Proteus: A self-designing range filter. In: Proceedings of the 2022 International Conference on Management of Data, pages 1670–1684
Zhang Huanchen, Lim Hyeontaek, Leis Viktor, Andersen David G, Kaminsky Michael, Keeton Kimberly, Pavlo Andrew (2018) Surf: Practical range query filtering with fast succinct tries. In: Proceedings of the 2018 International Conference on Management of Data, pages 323–336
Zhong Wenshao, Chen Chen, Wu Xingbo, Jiang Song (2021) Remix: Efficient range query for lsm-trees. In: 19th USENIX Conference on File and Storage Technologies (FAST 21), pages 51–64
Tang Chenlei, Wan Jiguang, Xie Changsheng (2022) Fencekv: Enabling efficient range query for key-value separation. IEEE Trans Parallel and Distributed Syst 33(12):3375–3386
Qin Mian, Zheng Qing, Lee Jason, Settlemyer Bradley, Wen Fei, Reddy Narasimha, Gratz Paul (2023) Kvrangedb: Range queries for a hash-based key-value device. ACM Trans Storage 19(3):1–21
Funding
This work was supported by Nature Science Foundation of Fujian Province under Grant No. 2021J01319, Fundamental Research Founds for the Central Universities of Huaqiao University under Grant No. ZQN-910, National Natural Science Foundation of China under Grant No. 61872086, Natural Science Foundation of Fujian Province of China under Grant No. 2022J06020, and the Young Top Talent of Young Eagle Program of Fujian Province, China under Grant No. F21E0011202B01.
Author information
Authors and Affiliations
Contributions
Conceptualization contributed by XL, YP, WF, HZ; methodology contributed by XL, YP, WF, HZ; investigation contributed by XL, YP, WF, HZ; formal analysis contributed by XL; software contributed by XL; validation contributed by XL; writing—original draft contributed by XL; supervision contributed by YP, ML; writing—review and editing contributed by YP, WF, HZ, ML; resources contributed by YP, ML.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare no Conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lin, X., Pan, Y., Feng, W. et al. RIOKV: reducing iterator overhead for efficient short-range query in LSM-tree-based key-value stores. J Supercomput 81, 343 (2025). https://doi.org/10.1007/s11227-024-06735-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s11227-024-06735-0