Skip to main content

RIOKV: reducing iterator overhead for efficient short-range query in LSM-tree-based key-value stores

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Short-range queries frequently occur in real-world scenarios. Traditional LSM-Tree-based key-value storage systems handle range query requests using an iterator mechanism. However, the data required for short-range queries typically comes from only a few iterators, and the construction of unnecessary iterators in the iterator mechanism degrades range query performance. To address this issue, we propose a MemTable Retention mechanism for Level 0 and an Iterator Reduction mechanism for other levels in LSM-Tree. The former retains the Immutable MemTable, which should have been persisted to Level 0, in memory to reduce the read overhead from Level 0 during short-range queries. The latter reduces the overhead of creating unnecessary iterators through the Level Filter Table. We developed a prototype system, RIOKV, based on LevelDB. Experimental results show that, under workloads with intensive short-range queries, RIOKV’s throughput is approximately 1.18\(-\)1.58 times that of LevelDB, 1.01\(-\)1.24 times that of RemixDB, and 1.06\(-\)1.46 times that of RocksDB. In workloads with mixed range queries and writes, RIOKV’s throughput is about 1.11\(-\)3.50 times that of LevelDB, 0.94\(-\)1.16 times that of RemixDB, and 1.28\(-\)3.24 times that of RocksDB.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Braun Lucas, Etter Thomas, Gasparis Georgios, Kaufmann Martin, Kossmann Donald, Widmer Daniel, Avitzur Aharon, Iliopoulos Anthony, Levy Eliezer, Liang Ning (2015) Proceedings of the 2015 Analytics in motion: High performance event-processing and real-time analytics in the same database. In: ACM SIGMOD International Conference on Management of Data 251–264

  2. Chen Guoqiang Jerry, Wiener Janet L, Iyer Shridhar, Jaiswal Anshul, Lei Ran, Simha Nikhil, Wang Wei, Wilfong Kevin, Williamson Tim, Yilmaz Serhat (2016) Realtime data processing at facebook. In: Proceedings of the 2016 In International Conference on Management of Data 1087–1098

  3. Goel Anil K, Pound Jeffrey, Auch Nathan, Bumbulis Peter, MacLean Scott, Färber Franz, Gropengiesser Francis, Mathis Christian, Bodner Thomas, Lehner Wolfgang (2015) Towards scalable real-time analytics: An architecture for scale-out of olxp workloads. Proceedings of the VLDB Endowment 8(12):1716–1727

    Article  Google Scholar 

  4. Yang Juncheng, Yue Yao, Rashmi KV (2021) A large-scale analysis of hundreds of in-memory key-value cache clusters at twitter. ACM Trans Storage (TOS) 17(3):1–35

    Article  MATH  Google Scholar 

  5. DeCandia Giuseppe, Hastorun Deniz, Jampani Madan, Kakulapati Gunavardhan, Lakshman Avinash, Pilchin Alex, Sivasubramanian Swaminathan, Vosshall Peter, Vogels Werner (2007) Dynamo: Amazon’s highly available key-value store. ACM SIGOPS Operating Syst Rev 41(6):205–220

    Article  Google Scholar 

  6. Debnath Biplob, Sengupta Sudipta, Li Jin (2011) Skimpystash: Ram space skimpy key-value store on flash-based storage. Proceedings of the 2011 ACM In SIGMOD International Conference on Management of data 56:25–36

    MATH  Google Scholar 

  7. Dinh Tien Tuan Anh, Wang Ji, Chen Gang, Liu Rui, Ooi Beng Chin, Tan Kian-Lee (2017) Blockbench: A framework for analyzing private blockchains. Proceedings of the 2017 In ACM international conference on management of data 56:1085–1100

    MATH  Google Scholar 

  8. O’Neil Patrick, Cheng Edward, Gawlick Dieter, O’Neil Elizabeth (1996) The log-structured merge-tree (lsm-tree). Acta Informatica 33:351–385

    Article  MATH  Google Scholar 

  9. Chang Fay, Dean Jeffrey, Ghemawat Sanjay, Hsieh Wilson C, Wallach Deborah A, Burrows Mike, Chandra Tushar, Fikes Andrew, Gruber Robert E (2008) Bigtable: A distributed storage system for structured data. ACM Trans Comput Syst (TOCS) 26(2):1–26

    Article  Google Scholar 

  10. LevelDB. 2021 https://github.com/google/leveldb,

  11. RocksDB. 2022 http://rocksdb.org/,

  12. Cassandra. http://cassandra.apache.org/, 2020

  13. HBase. 2020 http://hbase.apache.org/,

  14. Vora Mehul Nalin (2011) Hadoop-hbase for large-scale data. In: Proceedings of 2011 International Conference on Computer Science and Network Technology, volume 1, pages 601–605. IEEE

  15. Influx DB: (2022) Purpose-Built Open Source Time Series Database. https://www.influxdata.com/

  16. Siying Dong, Andrew Kryczka, Yanqin Jin, Michael Stumm (2021) Evolution of development priorities in key-value stores serving large-scale applications: The rocksdb experience. In: 19th USENIX Conference on File and Storage Technologies (FAST 21), pages 33–49,

  17. Berk Atikoglu, Yuehai Xu, Eitan Frachtenberg, Song Jiang, Mike Paleczny (2012) Workload analysis of a large-scale key-value store. In: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International conference on Measurement and Modeling of Computer Systems, 53–64,

  18. Cooper, Brian F, Silberstein Adam, Tam Erwin, Ramakrishnan Raghu, Sears Russell (2010)Benchmarking cloud serving systems with ycsb. In: Proceedings of the 1st ACM Symposium on Cloud Computing, 143–154

  19. Cao Zhichao, Dong Siying, Vemuri Sagar, Du David HC (2020) Characterizing, modeling, and benchmarking rocksdb key-value workloads at facebook. In: 18th USENIX Conference on File and Storage Technologies (FAST 20), 209–223

  20. Wang Ziwei, Zhong Zheng, Guo Jiarui, Wu Yuhan, Li Haoyu, Yang Tong, Tu Yaofeng, Zhang Huanchen, Cui Bin (2023) Rencoder: A space-time efficient range filter with local encoder. In: 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 2036–2049. IEEE

  21. Luo Siqiang, Chatterjee Subarna, Ketsetsidis Rafael, Dayan Niv, Qin Wilson, Idreos Stratos (2020) Rosetta: A robust space-time optimized range filter for key-value stores. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pages 2071–2086,

  22. Vaidya Kapil, Chatterjee Subarna, Knorr Eric, Mitzenmacher Michael, Idreos Stratos, Kraska Tim (2022) Snarf: a learning-enhanced range filter. Proceed VLDB Endowment 15(8):1632–1644

    Article  Google Scholar 

  23. Knorr, Eric R, Lemaire Baptiste, Lim Andrew, Luo Siqiang, Zhang Huanchen, Idreos Stratos, Mitzenmacher Michael (2022) Proteus: A self-designing range filter. In: Proceedings of the 2022 International Conference on Management of Data, pages 1670–1684

  24. Zhang Huanchen, Lim Hyeontaek, Leis Viktor, Andersen David G, Kaminsky Michael, Keeton Kimberly, Pavlo Andrew (2018) Surf: Practical range query filtering with fast succinct tries. In: Proceedings of the 2018 International Conference on Management of Data, pages 323–336

  25. Zhong Wenshao, Chen Chen, Wu Xingbo, Jiang Song (2021) Remix: Efficient range query for lsm-trees. In: 19th USENIX Conference on File and Storage Technologies (FAST 21), pages 51–64

  26. Tang Chenlei, Wan Jiguang, Xie Changsheng (2022) Fencekv: Enabling efficient range query for key-value separation. IEEE Trans Parallel and Distributed Syst 33(12):3375–3386

    Article  MATH  Google Scholar 

  27. Qin Mian, Zheng Qing, Lee Jason, Settlemyer Bradley, Wen Fei, Reddy Narasimha, Gratz Paul (2023) Kvrangedb: Range queries for a hash-based key-value device. ACM Trans Storage 19(3):1–21

    Article  Google Scholar 

Download references

Funding

This work was supported by Nature Science Foundation of Fujian Province under Grant No. 2021J01319, Fundamental Research Founds for the Central Universities of Huaqiao University under Grant No. ZQN-910, National Natural Science Foundation of China under Grant No. 61872086, Natural Science Foundation of Fujian Province of China under Grant No. 2022J06020, and the Young Top Talent of Young Eagle Program of Fujian Province, China under Grant No. F21E0011202B01.

Author information

Authors and Affiliations

Authors

Contributions

Conceptualization contributed by XL, YP, WF, HZ; methodology contributed by XL, YP, WF, HZ; investigation contributed by XL, YP, WF, HZ; formal analysis contributed by XL; software contributed by XL; validation contributed by XL; writing—original draft contributed by XL; supervision contributed by YP, ML; writing—review and editing contributed by YP, WF, HZ, ML; resources contributed by YP, ML.

Corresponding authors

Correspondence to Yubiao Pan or Mingwei Lin.

Ethics declarations

Conflict of interest

The authors declare no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, X., Pan, Y., Feng, W. et al. RIOKV: reducing iterator overhead for efficient short-range query in LSM-tree-based key-value stores. J Supercomput 81, 343 (2025). https://doi.org/10.1007/s11227-024-06735-0

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06735-0

Keywords