Abstract
LSM-based key-value stores have been leveraged in many state-of-the-art data-intensive applications as storage engines. As data volume scales up, a cost-efficient approach is to deploy these applications on hybrid cloud storage with hot/cold separation, which splits the LSM-tree into two parts and thus brings new challenges on how to split and how to close the significant performance gap between these two parts. Existing LSM-tree key-value stores mainly focus on the optimizations of local storage, which incurs sub-optimal performance when directly applied to hybrid storage.
In this paper, we present MirrorKV for efficient compaction and querying on hybrid cloud storage. First, based on the capacities of fast and slow cloud storage, MirrorKV vertically separates hot/cold data of different levels stored in different cloud storage with different compaction mechanisms. To avoid compaction in slow storage being the bottleneck of the write path, MirrorKV proposes a novel virtual split to only compact the metadata during the compaction, which postpones the actual compaction until it reaches deep enough levels. Second, to reduce accessing slow storage during querying, MirrorKV horizontally separates keys and values into two mirrored LSM-trees to differentiate caching priorities; the maintained tree structures preserve the data locality for efficient sequential reading without incurring the overhead of the traditional key-value separation solutions. Finally, MirrorKV leverages cached data to guide the compaction where the hot data is retained in the fast storage while the cold data is compacted to deeper levels in slow storage. Compared with RocksDB-cloud, MirrorKV achieves 2.4× higher random insertion throughput, 29% higher random read throughput, and 99% less compaction time.
- Matthias Brantner, Daniela Florescu, David Graf, Donald Kossmann, and Tim Kraska. 2008. Building a Database on S3. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (Vancouver, Canada) (SIGMOD '08). Association for Computing Machinery, New York, NY, USA, 251--264. https://doi.org/10.1145/1376616.1376645Google ScholarDigital Library
- Helen H. W. Chan, Yongkun Li, Patrick P. C. Lee, and Yinlong Xu. 2018. HashKV: Enabling Efficient Updates in KV Storage via Hashing. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 1007--1019. https://www.usenix.org/conference/atc18/presentation/chanGoogle Scholar
- Subarna Chatterjee, Meena Jagadeesan, Wilson Qin, and Stratos Idreos. 2021. Cosine: A Cloud-Cost Optimized Self-Designing Key-Value Storage Engine. Proc. VLDB Endow. 15, 1 (sep 2021), 112--126. https://doi.org/10.14778/3485450.3485461Google ScholarDigital Library
- Hao Chen, Chaoyi Ruan, Cheng Li, Xiaosong Ma, and Yinlong Xu. 2021. SpanDB: A Fast, Cost-Effective LSM-tree Based KV Store on Hybrid Storage. In 19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, 17--32. https://www.usenix.org/conference/fast21/presentation/chen-haoGoogle Scholar
- cockroachdb 2022. CockroachDB. https://www.cockroachlabs.com/product/.Google Scholar
- Alexander Conway, Abhishek Gupta, Vijay Chidambaram, Martin Farach-Colton, Richard Spillane, Amy Tai, and Rob Johnson. 2020. SplinterDB: Closing the Bandwidth Gap for NVMe Key-Value Stores. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 49--63. https://www.usenix.org/conference/atc20/presentation/conwayGoogle Scholar
- Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking Cloud Serving Systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing (Indianapolis, Indiana, USA) (SoCC '10). Association for Computing Machinery, New York, NY, USA, 143--154. https://doi.org/10.1145/1807128.1807152Google ScholarDigital Library
- Sudipto Das, Amr El Abbadi, and Divyakant Agrawal. 2009. ElasTraS: An Elastic Transactional Data Store in the Cloud. In Workshop on Hot Topics in Cloud Computing (HotCloud 09). USENIX Association, San Diego, CA. https://www.usenix.org/conference/hotcloud-09/elastras-elastic-transactional-data-store-cloudGoogle Scholar
- Niv Dayan, Manos Athanassoulis, and Stratos Idreos. 2017. Monkey: Optimal Navigable Key-Value Store. In Proceedings of the 2017 ACM International Conference on Management of Data (Chicago, Illinois, USA) (SIGMOD '17). Association for Computing Machinery, New York, NY, USA, 79--94. https://doi.org/10.1145/3035918.3064054Google ScholarDigital Library
- Niv Dayan and Stratos Idreos. 2018. Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores via Adaptive Removal of Superfluous Merging. In Proceedings of the 2018 International Conference on Management of Data (Houston, TX, USA) (SIGMOD '18). Association for Computing Machinery, New York, NY, USA, 505--520. https://doi.org/10.1145/3183713.3196927Google ScholarDigital Library
- Niv Dayan and Stratos Idreos. 2019. The Log-Structured Merge-Bush & the Wacky Continuum. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD '19). Association for Computing Machinery, New York, NY, USA, 449--466. https://doi.org/10.1145/3299869.3319903Google ScholarDigital Library
- Niv Dayan and Moshe Twitto. 2021. Chucky: A Succinct Cuckoo Filter for LSM-Tree. In Proceedings of the 2021 International Conference on Management of Data (Virtual Event, China) (SIGMOD '21). Association for Computing Machinery, New York, NY, USA, 365--378. https://doi.org/10.1145/3448016.3457273Google ScholarDigital Library
- Niv Dayan, Tamar Weiss, Shmuel Dashevsky, Michael Pan, Edward Bortnikov, and Moshe Twitto. 2022. Spooky: Granulating LSM-Tree Compactions Correctly. Proc. VLDB Endow. 15, 11 (jul 2022), 3071--3084. https://doi.org/10.14778/3551793.3551853Google ScholarDigital Library
- ebs 2022. Amazon Elastic Block Store. https://aws.amazon.com/ebs/.Google Scholar
- ebs_features 2022. Amazon EBS features. https://aws.amazon.com/ebs/features/?nc1=h_ls.Google Scholar
- efs 2022. Amazon Elastic File System. https://aws.amazon.com/efs/.Google Scholar
- flink 2022. Apache Flink - Stateful Computations over Data Streams. https://flink.apache.org/.Google Scholar
- gcs 2022. Google Cloud Storage. https://cloud.google.com/storage/.Google Scholar
- gfs 2022. Cloud Filestore. https://cloud.google.com/filestore.Google Scholar
- gpd 2022. Cloud Persistent Disk. https://cloud.google.com/persistent-disk.Google Scholar
- Shukai Han, Dejun Jiang, and Jin Xiong. 2020. SplitKV: Splitting IO Paths for Different Sized Key-Value Items with Advanced Storage Devices. In 12th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 20). USENIX Association. https://www.usenix.org/conference/hotstorage20/presentation/hanGoogle Scholar
- H. V. Jagadish, P. P. S. Narayan, S. Seshadri, S. Sudarshan, and Rama Kanneganti. 1997. Incremental Organization for Data Recording and Warehousing. In Proceedings of the 23rd International Conference on Very Large Data Bases (VLDB '97). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 16--25.Google ScholarDigital Library
- Olzhas Kaiyrakhmet, Songyi Lee, Beomseok Nam, Sam H. Noh, and Young ri Choi. 2019. SLM-DB: Single-Level Key-Value Store with Persistent Memory. In 17th USENIX Conference on File and Storage Technologies (FAST 19). USENIX Association, Boston, MA, 191--205. https://www.usenix.org/conference/fast19/presentation/kaiyrakhmetGoogle Scholar
- Sudarsun Kannan, Nitish Bhat, Ada Gavrilovska, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2018. Redesigning LSMs for Nonvolatile Memory with NoveLSM. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 993--1005. https://www.usenix.org/conference/atc18/presentation/kannanGoogle ScholarDigital Library
- Ana Klimovic, Heiner Litz, and Christos Kozyrakis. 2018. Selecta: Heterogeneous Cloud Storage Configuration for Data Analytics. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 759--773. https://www.usenix.org/conference/atc18/presentation/klimovic-selectaGoogle Scholar
- Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos Kozyrakis. 2018. Pocket: Elastic Ephemeral Storage for Serverless Analytics. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 427--444. https://www.usenix.org/conference/osdi18/presentation/klimovicGoogle ScholarDigital Library
- leveldbbench 2022. LevelDB bench tool. https://github.com/google/leveldb/blob/master/benchmarks/db_bench.cc.Google Scholar
- Haoyuan Li. 2018. Alluxio: A Virtual Distributed File System.Google Scholar
- Wenjie Li, Dejun Jiang, Jin Xiong, and Yungang Bao. 2020. HiLSM: An LSM-Based Key-Value Store for Hybrid NVM-SSD Storage Systems. In Proceedings of the 17th ACM International Conference on Computing Frontiers (Catania, Sicily, Italy) (CF '20). Association for Computing Machinery, New York, NY, USA, 208--216. https://doi.org/10.1145/3387902.3392621Google ScholarDigital Library
- Yongkun Li, Zhen Liu, Patrick P. C. Lee, Jiayu Wu, Yinlong Xu, Yi Wu, Liu Tang, Qi Liu, and Qiu Cui. 2021. Differentiated Key-Value Storage Management for Balanced I/O Performance. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 673--687. https://www.usenix.org/conference/atc21/presentation/li-yongkunGoogle Scholar
- Zhao Lucis Li, Chieh-Jan Mike Liang, Wenjia He, Lianjie Zhu, Wenjun Dai, Jin Jiang, and Guangzhong Sun. 2018. Metis: Robustly Tuning Tail Latencies of Cloud Systems. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 981--992. https://www.usenix.org/conference/atc18/presentation/li-zhaoGoogle Scholar
- Lanyue Lu, Thanumalayan Sankaranarayana Pillai, Hariharan Gopalakrishnan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2017. WiscKey: Separating Keys from Values in SSD-Conscious Storage. ACM Trans. Storage 13, 1, Article 5 (March 2017), 28 pages. https://doi.org/10.1145/3033273Google ScholarDigital Library
- Chen Luo and Michael J. Carey. 2019. LSM-Based Storage Techniques: A Survey. The VLDB Journal 29, 1 (jul 2019), 393--418. https://doi.org/10.1007/s00778-019-00555-yGoogle ScholarDigital Library
- Chen Luo and Michael J. Carey. 2019. On Performance Stability in LSM-Based Storage Systems. Proc. VLDB Endow. 13, 4 (dec 2019), 449--462. https://doi.org/10.14778/3372716.3372719Google ScholarDigital Library
- Siqiang Luo, Subarna Chatterjee, Rafael Ketsetsidis, Niv Dayan, Wilson Qin, and Stratos Idreos. 2020. Rosetta: A Robust Space-Time Optimized Range Filter for Key-Value Stores. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (Portland, OR, USA) (SIGMOD '20). Association for Computing Machinery, New York, NY, USA, 2071--2086. https://doi.org/10.1145/3318464.3389731Google ScholarDigital Library
- mongodb 2022. MongoDB. https://www.mongodb.com/.Google Scholar
- Bernhard Mößner, Christian Riegger, Arthur Bernhardt, and Ilia Petrov. 2023. bloomRF: On Performing Range-Queries in Bloom-Filters with Piecewise-Monotone Hash Functions and Prefix Hashing. In Proceedings 26th International Conference on Extending Database Technology, EDBT 2023, Ioannina, Greece, March 28--31, 2023, Julia Stoyanovich, Jens Teubner, Nikos Mamoulis, Evaggelia Pitoura, Jan Mühlig, Katja Hose, Sourav S. Bhowmick, and Matteo Lissandrini (Eds.). OpenProceedings.org, 131--143. https://doi.org/10.48786/edbt.2023.11Google ScholarCross Ref
- myrocks 2022. A RocksDB storage engine with MySQL. http://myrocks.io/.Google Scholar
- Patrick O'Neil, Edward Cheng, Dieter Gawlick, and Elizabeth O'Neil. 1996. The Log-Structured Merge-Tree (LSM-Tree). Acta Inf. 33, 4 (June 1996), 351--385. https://doi.org/10.1007/s002360050048Google ScholarDigital Library
- presto 2022. Presto - Distributed SQL Query Engine for Big Data. https://prestodb.io/.Google Scholar
- Pandian Raju, Rohan Kadekodi, Vijay Chidambaram, and Ittai Abraham. 2017. PebblesDB: Building Key-Value Stores Using Fragmented Log-Structured Merge Trees. In Proceedings of the 26th Symposium on Operating Systems Principles (Shanghai, China) (SOSP '17). Association for Computing Machinery, New York, NY, USA, 497--514. https://doi.org/10.1145/3132747.3132765Google ScholarDigital Library
- Kai Ren, Qing Zheng, Joy Arulraj, and Garth Gibson. 2017. SlimDB: A Space-Efficient Key-Value Storage Engine for Semi-Sorted Data. Proc. VLDB Endow. 10, 13 (Sept. 2017), 2037--2048. https://doi.org/10.14778/3151106.3151108Google ScholarDigital Library
- rocksdbcache 2022. RocksDB Block Cache. https://github.com/facebook/rocksdb/wiki/Block-Cache.Google Scholar
- rocksdbcloud 2022. RocksDB-Cloud: A Key-Value Store for Cloud Applications. https://github.com/rockset/rocksdb-cloud.Google Scholar
- s3 2022. Amazon S3. https://aws.amazon.com/s3/.Google Scholar
- Subhadeep Sarkar, Dimitris Staratzis, Ziehen Zhu, and Manos Athanassoulis. 2021. Constructing and Analyzing the LSM Compaction Design Space. Proc. VLDB Endow. 14, 11 (jul 2021), 2216--2229. https://doi.org/10.14778/3476249.3476274Google ScholarDigital Library
- spark 2022. Apache Spark - Lightning-fast unified analytics engine. https://spark.apache.org/.Google Scholar
- Yuzhe Tang, Arun Iyengar, Wei Tan, Liana Fong, Ling Liu, and Balaji Palanisamy. 2015. Deferred Lightweight Indexing for Log-Structured Key-Value Stores. In 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. 11--20. https://doi.org/10.1109/CCGrid.2015.150Google ScholarDigital Library
- tidb 2022. PingCAP. https://pingcap.com/products/tidb.Google Scholar
- Kapil Vaidya, Subarna Chatterjee, Eric Knorr, Michael Mitzenmacher, Stratos Idreos, and Tim Kraska. 2022. SNARF: A Learning-Enhanced Range Filter. Proc. VLDB Endow. 15, 8 (apr 2022), 1632--1644. https://doi.org/10.14778/3529337.3529347Google ScholarDigital Library
- Ziwei Wang, Zheng Zhong, Jiarui Guo, Yuhan Wu, Haoyu Li, Tong Yang, Yaofeng Tu, Huanchen Zhang, and Bin Cui. [n. d.]. REncoder: A Space-Time Efficient Range Filter with Local Encoder. ([n. d.]).Google Scholar
- wisckeygithub 2022. Implementation of WiscKey. https://github.com/cld378632668/Wisckey_SeparateKVStorage.Google Scholar
- Xingbo Wu, Yuehai Xu, Zili Shao, and Song Jiang. 2015. LSM-trie: An LSM-tree-based Ultra-Large Key-Value Store for Small Data Items. In 2015 USENIX Annual Technical Conference (USENIX ATC 15). USENIX Association, Santa Clara, CA, 71--82. https://www.usenix.org/conference/atc15/technical-session/presentation/wuGoogle ScholarDigital Library
- Peng Xu, Nannan Zhao, Jiguang Wan, Wei Liu, Shuning Chen, Yuanhui Zhou, Hadeel Albahar, Hanyang Liu, Liu Tang, and Changsheng Xie. 2021. Building A Fast and Efficient LSM-tree Store by Integrating Local Storage with Cloud Storage. In 2021 IEEE International Conference on Cluster Computing (CLUSTER). 125--134. https://doi.org/10.1109/Cluster48925.2021.00032Google ScholarCross Ref
- Ting Yao, Zhihu Tan, Jiguang Wan, Ping Huang, Yiwen Zhang, Changsheng Xie, and Xubin He. 2019. SEALDB: An Efficient LSM-tree Based KV Store on SMR Drives with Sets and Dynamic Bands. IEEE Transactions on Parallel and Distributed Systems 30, 11 (2019), 2595--2607. https://doi.org/10.1109/TPDS.2019.2918219Google ScholarDigital Library
- Ting Yao, Jiguang Wan, Ping Huang, Yiwen Zhang, Zhiwen Liu, Changsheng Xie, and Xubin He. 2019. GearDB: A GC-free Key-Value Store on HM-SMR Drives with Gear Compaction. In 17th USENIX Conference on File and Storage Technologies (FAST 19) (Boston, MA, USA) (FAST'19). USENIX Association, Boston, MA, 159--171. https://www.usenix.org/conference/fast19/presentation/yaoGoogle Scholar
- Ting Yao, Yiwen Zhang, Jiguang Wan, Qiu Cui, Liu Tang, Hong Jiang, Changsheng Xie, and Xubin He. 2020. MatrixKV: Reducing Write Stalls and Write Amplification in LSM-tree Based KV Stores with Matrix Container in NVM. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 17--31. https://www.usenix.org/conference/atc20/presentation/yaoGoogle Scholar
- Hobin Yoon, Juncheng Yang, Sveinn Fannar Kristjansson, Steinn E. Sigurdarson, Ymir Vigfusson, and Ada Gavrilovska. 2018. Mutant: Balancing Storage Cost and Latency in LSM-Tree Data Stores. In Proceedings of the ACM Symposium on Cloud Computing (Carlsbad, CA, USA) (SoCC '18). Association for Computing Machinery, New York, NY, USA, 162--173. https://doi.org/10.1145/3267809.3267846Google ScholarDigital Library
- Huanchen Zhang, Hyeontaek Lim, Viktor Leis, David G. Andersen, Michael Kaminsky, Kimberly Keeton, and Andrew Pavlo. 2018. SuRF: Practical Range Query Filtering with Fast Succinct Tries. In Proceedings of the 2018 International Conference on Management of Data (Houston, TX, USA) (SIGMOD '18). Association for Computing Machinery, New York, NY, USA, 323--336. https://doi.org/10.1145/3183713.3196931Google ScholarDigital Library
- Q. Zhang, Y. Li, P. P. C. Lee, Y. Xu, Q. Cui, and L. Tang. 2020. UniKV: Toward High-Performance and Scalable KV Storage in Mixed Workloads via Unified Indexing. In 2020 IEEE 36th International Conference on Data Engineering (ICDE). 313--324. https://doi.org/10.1109/ICDE48307.2020.00034Google ScholarCross Ref
- zhichao Cao, Siying Dong, Sagar Vemuri, and David H.C. Du. 2020. Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook. In 18th USENIX Conference on File and Storage Technologies (FAST 20). USENIX Association, Santa Clara, CA, 209--223. https://www.usenix.org/conference/fast20/presentation/cao-zhichaoGoogle Scholar
- Jia Zou, Arun Iyengar, and Chris Jermaine. 2019. Pangea: monolithic distributed storage for data analytics. Proceedings of the VLDB Endowment 12 (02 2019), 681--694. https://doi.org/10.14778/3311880.3311885Google ScholarDigital Library
Index Terms
- MirrorKV: An Efficient Key-Value Store on Hybrid Cloud Storage with Balanced Performance of Compaction and Querying
Recommendations
An Efficient Memory-Mapped Key-Value Store for Flash Storage
SoCC '18: Proceedings of the ACM Symposium on Cloud ComputingPersistent key-value stores have emerged as a main component in the data access path of modern data processing systems. However, they exhibit high CPU and I/O overhead. Today, due to power limitations it is important to reduce CPU overheads for data ...
LSM-tree managed storage for large-scale key-value store
SoCC '17: Proceedings of the 2017 Symposium on Cloud ComputingKey-value stores are increasingly adopting LSM-trees as their enabling data structure in the backend storage, and persisting their clustered data through a file system. A file system is expected to not only provide file/directory abstraction to organize ...
Building Efficient Key-Value Stores via a Lightweight Compaction Tree
Special Issue on MSST 2017 and Regular PapersLog-Structure Merge tree (LSM-tree) has been one of the mainstream indexes in key-value systems supporting a variety of write-intensive Internet applications in today’s data centers. However, the performance of LSM-tree is seriously hampered by ...
Comments