A High-Performance Hybrid Index Framework Supporting Inserts for Static Learned Indexes

Abstract

The learned index is a new index structure that uses a trained model to directly predict the position of a key and thus has high query performance. However, static learned indexes cannot handle insert operations. Although static PGM-index uses a dynamic data structure to support inserts, it faces a serious read amplification problem under read-write workloads, as the inefficient lookup process of the buffers diminishes the learned indexes. Besides, this structure also leads to periodic retraining of the internal PGM-indexes because the buffers and the learned indexes are strongly coupled, which is unacceptable for those static learned indexes that need tuning. Obviously, this structure is not an ideal general framework. In this paper, we propose a two-layer Hybrid Index Framework (HIF) to address such issues. Specifically, the dynamic layer is used as a buffer for inserts, and the static layer consisting of static learned indexes is used for lookups only. HIF effectively alleviates read amplification by searching the static layer directly. And with this hierarchical structure, HIF isolates learned indexes from insert operations. Thus HIF can completely avoid the retraining of the learned indexes by transformation strategy from the dynamic layer to the static layer. Moreover, we provide a self-tuning algorithm for the learned indexes that cannot be built in a single pass over the data, allowing them to be applied to dynamic workloads with low training overhead. We have conducted experiments using multiple datasets and workloads and the results show that on average, three HIF-based static learned indexes, HLI, PGM, and RMI, achieve up to 1.8 $\times$, 1.7 $\times$, and 1.5 $\times$ higher throughput than the original dynamic PGM-index for insert ratio below 70%.