Niv Dayan
Moshe Twitto
ABSTRACT
Modern key-value stores typically rely on an LSM-tree in storage (SSD) to handle writes and Bloom filters in memory (DRAM) to optimize reads. With ongoing advances in SSD technology shrinking the performance gap between storage and memory devices, the Bloom filters are now emerging as a performance bottleneck.
We propose Chucky, a new design that replaces the multiple Bloom filters by a single Cuckoo filter that maps each data entry to an auxiliary address of its location within the LSM-tree. We show that while such a design entails fewer memory accesses than with Bloom filters, its false positive rate off the bat is higher. The reason is that the auxiliary addresses occupy bits that would otherwise be used as parts of the Cuckoo filter's fingerprints. To address this, we harness techniques from information theory to succinctly encode the auxiliary addresses so that the fingerprints can stay large. As a result, Chucky achieves the best of both worlds: a modest access cost and a low false positive rate at the same time.
Supplemental Material
- Ahmad, M. Y., and Kemme, B. Compaction management in distributed key-value datastores. PVLDB 8, 8 (2015), 850--861.Google Scholar
Digital Library
- Alsubaiee, S., Altowim, Y., Altwaijry, H., Behm, A., Borkar, V. R., Bu, Y., Carey, M. J., Cetindil, I., Cheelangi, M., Faraaz, K., Gabrielova, E., Grover, R., Heilbron, Z., Kim, Y.-S., Li, C., Li, G., Ok, J. M., Onose, N., Pirzadeh, P., Tsotras, V. J., Vernica, R., Wen, J., and Westmann, T. AsterixDB: A Scalable, Open Source BDMS. PVLDB 7, 14 (2014), 1905--1916.Google ScholarDigital Library
- Apache. Cassandra. http://cassandra.apache.org.Google Scholar
- Apache. HBase. http://hbase.apache.org/.Google Scholar
- Athanassoulis, M., and Idreos, S. Design Tradeoffs of Data Access Methods. SIGMOD (2016).Google Scholar
- Athanassoulis, M., Kester, M. S., Maas, L. M., Stoica, R., Idreos, S., Ailamaki, A., and Callaghan, M. Designing Access Methods: The RUM Conjecture. EDBT (2016).Google Scholar
- Balmau, O., Didona, D., Guerraoui, R., Zwaenepoel, W., Yuan, H., Arora, A., Gupta, K., and Konka, P. TRIAD: Creating Synergies Between Memory, Disk and Log in Log Structured Key-Value Stores. USENIX ATC (2017).Google Scholar
- Balmau, O., Dinu, F., Zwaenepoel, W., Gupta, K., Chandhiramoorthi, R., and Didona, D. $$SILK$$: Preventing latency spikes in log-structured merge key-value stores. In USENIX ATC (2019).Google Scholar
- Bender, M. A., Farach-Colton, M., Johnson, R., Kraner, R., Kuszmaul, B. C., Medjedovic, D., Montes, P., Shetty, P., Spillane, R. P., and Zadok, E. Don't Thrash: How to Cache Your Hash on Flash. PVLDB 5, 11 (2012), 1627--1637.Google ScholarDigital Library
- Bjørling, M., Bonnet, P., Bouganim, L., and Dayan, N. The Necessary Death of the Block Device Interface. CIDR (2013).Google Scholar
- Bloom, B. H. Space/Time Trade-offs in Hash Coding with Allowable Errors. CACM 13, 7 (1970), 422--426.Google ScholarDigital Library
- Bonnet, P., and Dayan, N. Solid-state storage device flash translation layer, 2017. US Patent App. 15/056,381.Google Scholar
- Bonomi, F., Mitzenmacher, M., Panigrahy, R., Singh, S., and Varghese, G. An improved construction for counting bloom filters. In European Symposium on Algorithms (2006).Google ScholarDigital Library
- Bortnikov, E., Braginsky, A., Hillel, E., Keidar, I., and Sheffi, G. Accordion: Better Memory Organization for LSM Key-Value Stores. PVLDB 11, 12 (2018), 1863--1875.Google Scholar
- Breslow, A. D., and Jayasena, N. S. Morton filters: faster, space-efficient cuckoo filters via biasing, compression, and decoupled logical sparsity. In VLDB (2018).Google ScholarDigital Library
- Broder, A. Z., and Mitzenmacher, M. Network Applications of Bloom Filters: A Survey. Internet Mathematics 1 (2002), 636--646.Google Scholar
- Canim, M., Mihaila, G. A., Bhattacharjee, B., Ross, K. A., and Lang, C. A. SSD Bufferpool Extensions for Database Systems. PVLDB 3, 1--2 (2010), 1435--1446.Google Scholar
- Chan, H. H. W., Li, Y., Lee, P. P. C., and Xu, Y. HashKV: Enabling Efficient Updates in KV Storage via Hashing. ATC (2018).Google ScholarDigital Library
- Charles, D., and Chellapilla, K. Bloomier filters: A second look. In European Symposium on Algorithms (2008).Google ScholarDigital Library
- Chazelle, B., Kilian, J., Rubinfeld, R., and Tal, A. The bloomier filter: an efficient data structure for static support lookup tables. In Symposium on Discrete Algorithms (2004).Google Scholar
- Chen, G. J., Wiener, J. L., Iyer, S., Jaiswal, A., Lei, R., Simha, N., Wang, W., Wilfong, K., Williamson, T., and Yilmaz, S. Realtime data processing at facebook. In SIGMOD (2016).Google Scholar
- Chen, H., Liao, L., Jin, H., and Wu, J. The dynamic cuckoo filter. In IEEE ICNP (2017).Google ScholarCross Ref
- Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., and Sears, R. Benchmarking cloud serving systems with YCSB. SoCC (2010).Google Scholar
- Dai, Y., Xu, Y., Ganesan, A., Alagappan, R., Kroth, B., Arpaci-Dusseau, A., and Arpaci-Dusseau, R. From wisckey to bourbon: A learned index for log-structured merge trees. In USENIX OSDI (2020).Google Scholar
- Dayan, N., Athanassoulis, M., and Idreos, S. Monkey: Optimal Navigable Key-Value Store. SIGMOD (2017).Google ScholarDigital Library
- Dayan, N., Athanassoulis, M., and Idreos, S. Optimal Bloom Filters and Adaptive Merging for LSM-Trees. TODS 43, 4 (2018), 16:1--16:48.Google Scholar
- Dayan, N., Bonnet, P., and Idreos, S. GeckoFTL: Scalable Flash Translation Techniques For Very Large Flash Devices. SIGMOD (2016).Google ScholarDigital Library
- Dayan, N., and Idreos, S. Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores via Adaptive Removal of Superfluous Merging. SIGMOD (2018).Google Scholar
- Dayan, N., and Idreos, S. The log-structured merge-bush & the wacky continuum. In SIGMOD (2019).Google ScholarDigital Library
- Dayan, N., Svendsen, M. K., Bjorling, M., Bonnet, P., and Bouganim, L. Eagletree: exploring the design space of ssd-based algorithms. VLDB (2013).Google ScholarDigital Library
- Debnath, B., Sengupta, S., Li, J., Lilja, D. J., and Du, D. H. Bloomflash: Bloom filter on flash-based storage. In ICDCS (2011).Google ScholarDigital Library
- Deeds, K., Hentschel, B., and Idreos, S. Stacked filters: learning to filter by structure. PVLDB (2020).Google ScholarDigital Library
- Dillinger, P. C., and Manolios, P. Bloom Filters in Probabilistic Verification. Formal Methods in Computer-Aided Design (2004).Google Scholar
- Dinh, T. T. A., Wang, J., Chen, G., Liu, R., Ooi, B. C., and Tan, K.-L. Blockbench: A framework for analyzing private blockchains. In SIGMOD (2017).Google ScholarDigital Library
- Dong, S., Callaghan, M., Galanis, L., Borthakur, D., Savor, T., and Strum, M. Optimizing Space Amplification in RocksDB. CIDR (2017).Google Scholar
- Duda, J., Tahboub, K., Gadgil, N. J., and Delp, E. J. The use of asymmetric numeral systems as an accurate replacement for huffman coding. In Picture Coding Symposium (PCS) (2015).Google ScholarCross Ref
- Eisenman, A., Gardner, D., AbdelRahman, I., Axboe, J., Dong, S., Hazelwood, K., Petersen, C., Cidon, A., and Katti, S. Reducing dram footprint with nvm in facebook. In EuroSys (2018).Google ScholarDigital Library
- Facebook. RocksDB. https://github.com/facebook/rocksdb.Google Scholar
- Fan, B., Andersen, D. G., Kaminsky, M., and Mitzenmacher, M. Cuckoo Filter: Practically Better Than Bloom. CoNEXT (2014).Google ScholarDigital Library
- Fan, L., Cao, P., Almeida, J., and Broder, A. Z. Summary cache: A scalable wide-area Web cache sharing protocol. IEEE/ACM Transactions on Networking 8, 3 (2000), 281--293.Google Scholar
- Gilad, E., Bortnikov, E., Braginsky, A., Gottesman, Y., Hillel, E., Keidar, I., Moscovici, N., and Shahout, R. Evendb: Optimizing key-value storage for spatial locality. In EuroSys (2020).Google ScholarDigital Library
- Golan-Gueta, G., Bortnikov, E., Hillel, E., and Keidar, I. Scaling Concurrent Log-Structured Data Stores. EuroSys (2015).Google Scholar
- Golomb, S. Run-length encodings. IEEE transactions on information theory (1966).Google Scholar
- Graf, T. M., and Lemire, D. Xor filters: Faster and smaller than bloom and cuckoo filters. Journal of Experimental Algorithmics (2020).Google ScholarDigital Library
- Huang, G., Cheng, X., Wang, J., Wang, Y., He, D., Zhang, T., Li, F., Wang, S., Cao, W., and Li, Q. X-engine: An optimized storage engine for large-scale e-commerce transaction processing. In SIGMOD (2019).Google ScholarDigital Library
- Huffman, D. A. A Method for the Construction of Minimum-Redundancy Codes. Proceedings of the IRE 40, 9 (1952), 1098--1101.Google ScholarCross Ref
- Idreos, S., Athanassoulis, M., Dayan, N., Guo, D., Kester, M. S., Maas, L., and Zoumpatianos, K. Past and future steps for adaptive storage data systems: From shallow to deep adaptivity. In Real-Time Business Intelligence and Analytics. Springer, 2015.Google Scholar
- Idreos, S., and Callaghan, M. Key-value storage engines. In SIGMOD (2020).Google ScholarDigital Library
- Idreos, S., and Dayan, N. File management with log-structured merge bush, 2020. US Patent App. 16/963,411.Google Scholar
- Idreos, S., and Dayan, N. Key-value stores with optimized merge policies and optimized lsm-tree structures, 2020. US Patent App. 16/963,411.Google Scholar
- Idreos, S., Dayan, N., and Athanassoulis, M. Optimized navigable key-value store, 2020. US Patent App. 16/433,075.Google Scholar
- Idreos, S., Dayan, N., Qin, W., Akmanalp, M., Hilgard, S., Ross, A., Lennon, J., Jain, V., Gupta, H., Li, D., et al. Learning key-value store design. arXiv preprint arXiv:1907.05443 (2019).Google Scholar
- Idreos, S., Dayan, N., Qin, W., Akmanalp, M., Hilgard, S., Ross, A., Lennon, J., Jain, V., Gupta, H., Li, D., and Zhu, Z. Design continuums and the path toward self-designing key-value stores that know and learn. In CIDR (2019).Google Scholar
- Idreos, S., and Kraska, T. From auto-tuning one size fits all to self-designed and learned data-intensive systems. In SIGMOD (2019).Google ScholarDigital Library
- Idreos, S., Zoumpatianos, K., Athanassoulis, M., Dayan, N., Hentschel, B., Kester, M. S., Guo, D., Maas, L. M., Qin, W., Wasay, A., and Sun, Y. The Periodic Table of Data Structures. IEEE DEBULL 41, 3 (2018), 64--75.Google Scholar
- Idreos, S., Zoumpatianos, K., Chatterjee, S., Qin, W., Wasay, A., Hentschel, B., Kester, M., Dayan, N., Guo, D., Kang, M., et al. Learning data structure alchemy. IEEE DEBULL (2019).Google Scholar
- Idreos, S., Zoumpatianos, K., Hentschel, B., Kester, M. S., and Guo, D. The Data Calculator: Data Structure Design and Cost Synthesis from First Principles and Learned Cost Models. SIGMOD (2018).Google ScholarDigital Library
- Im, J., Bae, J., Chung, C., Lee, S., et al. Pink: High-speed in-storage key-value store with bounded tails. In USENIX ATC (2020).Google Scholar
- Jagadish, H. V., Narayan, P. P. S., Seshadri, S., Sudarshan, S., and Kanneganti, R. Incremental Organization for Data Recording and Warehousing. VLDB (1997).Google ScholarDigital Library
- Kirsch, A., and Mitzenmacher, M. Less hashing, same performance: Building a better Bloom filter. Random Structures & Algorithms 33, 2 (2008), 187--218.Google ScholarCross Ref
- Kondylakis, H., Dayan, N., Zoumpatianos, K., and Palpanas, T. Coconut: A scalable bottom-up approach for building data series indexes. VLDB 11, 6 (2018), 677--690.Google Scholar
- Kondylakis, H., Dayan, N., Zoumpatianos, K., and Palpanas, T. Coconut palm: Static and streaming data series exploration now in your palm. In SIGMOD (2019).Google ScholarDigital Library
- Kondylakis, H., Dayan, N., Zoumpatianos, K., and Palpanas, T. Coconut: sortable summarizations for scalable indexes over static and streaming data series. VLDBJ (2019).Google Scholar
- Kraft, L. G. A device for quantizing, grouping, and coding amplitude-modulated pulses. PhD thesis, MIT, 1949.Google Scholar
- Kraska, T., Beutel, A., Chi, E. H., Dean, J., and Polyzotis, N. The Case for Learned Index Structures. SIGMOD (2018).Google Scholar
- Lang, H., Neumann, T., Kemper, A., and Boncz, P. Performance-optimal filtering: Bloom overtakes cuckoo at high throughput. In VLDB (2019).Google ScholarDigital Library
- Lu, G., Debnath, B., and Du, D. H. C. A Forest-structured Bloom Filter with flash memory. MSST (2011).Google Scholar
- Lu, L., Pillai, T. S., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. WiscKey: Separating Keys from Values in SSD-conscious Storage. FAST (2016).Google ScholarDigital Library
- Luo, C., and Carey, M. J. On performance stability in lsm-based storage systems. In VLDB (2019).Google ScholarDigital Library
- Luo, C., and Carey, M. J. Lsm-based storage techniques: a survey. The VLDB Journal (2020).Google Scholar
- Luo, C., Tözün, P., Tian, Y., Barber, R., Raman, V., and Sidle, R. Umzi: Unified multi-zone indexing for large-scale htap. In EDBT (2019).Google Scholar
- Luo, L., Guo, D., Ma, R. T., Rottenstreich, O., and Luo, X. Optimizing bloom filter: Challenges, solutions, and comparisons. IEEE Commun. Surv. Tutor. (2018).Google Scholar
- Luo, S., Chatterjee, S., Ketsetsidis, R., Dayan, N., Qin, W., and Idreos, S. Rosetta: A robust space-time optimized range filter for key-value stores. In SIGMOD (2020).Google ScholarDigital Library
- Matsunobu, Y., Dong, S., and Lee, H. Myrocks: Lsm-tree database storage engine serving facebook's social graph. VLDB (2020).Google ScholarDigital Library
- McMillan, B. Two inequalities implied by unique decipherability. IRE Transactions on Information Theory (1956).Google ScholarCross Ref
- Mei, F., Cao, Q., Jiang, H., and Li, J. Sifrdb: A unified solution for write-optimized key-value stores in large datacenter. In ACM SOCC (2018).Google ScholarDigital Library
- Mitzenmacher, M. Compressed bloom filters. IEEE/ACM Transactions on Networking (2002).Google ScholarDigital Library
- Mitzenmacher, M., Pontarelli, S., and Reviriego, P. Adaptive cuckoo filters. In SIAM ALENEX (2018).Google ScholarCross Ref
- O'Neil, P. E., Cheng, E., Gawlick, D., and O'Neil, E. J. The log-structured merge-tree (LSM-tree). Acta Informatica 33, 4 (1996), 351--385.Google ScholarDigital Library
- Pagh, A., Pagh, R., and Rao, S. S. An optimal bloom filter replacement. In SODA (2005).Google ScholarDigital Library
- Pandey, P., Bender, M. A., Johnson, R., and Patro, R. A general-purpose counting filter: Making every bit count. In SIGMOD (2017).Google ScholarDigital Library
- Pasco, R. C. Source coding algorithms for fast data compression. PhD thesis, Stanford University CA, 1976.Google ScholarDigital Library
- Polychroniou, O., and Ross, K. A. Vectorized Bloom filters for advanced SIMD processors. DAMON (2014).Google Scholar
- Putze, F., Sanders, P., and Singler, J. Cache-, hash-, and space-efficient bloom filters. Journal of Experimental Algorithmics (JEA) (2010).Google Scholar
- Raju, P., Kadekodi, R., Chidambaram, V., and Abraham, I. PebblesDB: Building Key-Value Stores using Fragmented Log-Structured Merge Trees. SOSP (2017).Google ScholarDigital Library
- Ren, K., Zheng, Q., Arulraj, J., and Gibson, G. SlimDB: A Space-Efficient Key-Value Storage Engine For Semi-Sorted Data. PVLDB 10, 13 (2017), 2037--2048.Google ScholarDigital Library
- Rissanen, J., and Langdon, G. G. Arithmetic coding. IBM Journal of research and development (1979).Google Scholar
- Rothenberg, C. E., Macapuna, C., Verdi, F., and Magalhaes, M. The deletable Bloom filter: a new member of the Bloom family. IEEE Communications Letters 14, 6 (jun 2010), 557--559.Google ScholarDigital Library
- Rottenstreich, O., Kanizo, Y., and Keslassy, I. The variable-increment counting bloom filter. IEEE/ACM Transactions on Networking (2013).Google Scholar
- Sarkar, S., Papon, T. I., Staratzis, D., and Athanassoulis, M. Lethe: A tunable delete-aware lsm engine. In SIGMOD (2020).Google ScholarDigital Library
- Sears, R., and Ramakrishnan, R. bLSM: A General Purpose Log Structured Merge Tree. SIGMOD (2012).Google ScholarDigital Library
- Shetty, P., Spillane, R. P., Malpani, R., Andrews, B., Seyster, J., and Zadok, E. Building Workload-Independent Storage with VT-trees. FAST (2013).Google ScholarDigital Library
- Tarkoma, S., Rothenberg, C. E., and Lagerspetz, E. Theory and Practice of Bloom Filters for Distributed Systems. IEEE Communications Surveys & Tutorials 14, 1 (2012), 131--155.Google ScholarCross Ref
- Thonangi, R., and Yang, J. On Log-Structured Merge for Solid-State Drives. ICDE (2017).Google Scholar
- Vincc on, T., Hardock, S., Riegger, C., Oppermann, J., Koch, A., and Petrov, I. Noftl-kv: Tackling write-amplification on kv-stores with native storage management. In EDBT (2018).Google Scholar
- Wang, M., Zhou, M., Shi, S., and Qian, C. Vacuum filters: more space-efficient and faster replacement for bloom and cuckoo filters. In VLDB (2019).Google ScholarDigital Library
- Wang, P., Sun, G., Jiang, S., Ouyang, J., Lin, S., Zhang, C., and Cong, J. An Efficient Design and Implementation of LSM-Tree based Key-Value Store on Open-Channel SSD. EuroSys (2014).Google Scholar
- Wu, X., Xu, Y., Shao, Z., and Jiang, S. LSM-trie: An LSM-tree-based Ultra-Large Key-Value Store for Small Data Items. USENIX ATC (2015).Google Scholar
- Yao, T., Wan, J., Huang, P., He, X., Wu, F., and Xie, C. Building Efficient Key-Value Stores via a Lightweight Compaction Tree. TOS 13, 4 (2017), 29:1--29:28.Google ScholarDigital Library
- Zhang, H., Lim, H., Leis, V., Andersen, D. G., Kaminsky, M., Keeton, K., and Pavlo, A. SuRF: Practical Range Query Filtering with Fast Succinct Tries. SIGMOD (2018).Google ScholarDigital Library
- Zhang, T., Wang, J., Cheng, X., Xu, H., Yu, N., Huang, G., Zhang, T., He, D., Li, F., Cao, W., et al. Fpga-accelerated compactions for lsm-based key-value store. In USENIX FAST (2020).Google Scholar
Index Terms
- Chucky: A Succinct Cuckoo Filter for LSM-Tree
Recommendations
MDCF: Multiple Dynamic Cuckoo Filters for LSM-Tree
Algorithms and Architectures for Parallel ProcessingAbstractAs a write-optimized data structure, the Log-Structured Merge-tree (LSM-tree) based storage engine, which maintains data in a leveled structure on disk, is widely used in Key-Value (KV) storage systems. Meanwhile, the leveled design also makes it ...
LSM-tree managed storage for large-scale key-value store
SoCC '17: Proceedings of the 2017 Symposium on Cloud ComputingKey-value stores are increasingly adopting LSM-trees as their enabling data structure in the backend storage, and persisting their clustered data through a file system. A file system is expected to not only provide file/directory abstraction to organize ...
FlatLSM: Write-Optimized LSM-Tree for PM-Based KV Stores
The Log-Structured Merge Tree (LSM-Tree) is widely used in key-value (KV) stores because of its excwrite performance. But LSM-Tree-based KV stores still have the overhead of write-ahead log and write stall caused by slow L0 flush and L0-L1 compaction. New ...
Comments