A Memory Optimization Method for High Cardinality Based on InfluxDB

The time series databases support various monitoring systems and data analysis systems. However, high cardinality may be causing memory issues. when handling a heavy workload, time series databases consume a lot of memory, resulting in a drop in read and write performance. In order to solve this problem, a memory optimization method for high cardinality based on InfluxDB is proposed. This method first reduces the number of time series data in the buffer, that is, by reducing the number of time series data decompressed by InfluxDB from the disk, thereby reducing the memory overhead required for buffering temporary data. Secondly, it delays the reading of non-essential data during the current query phase. By utilizing the data skewness between timelines, the reading of non-essential data blocks can be postponed, reducing the number of timelines present in memory at any given moment and thereby decreasing memory overhead. Experiments show that in high cardinality scenarios with a concurrency of 100,000, the optimized InfluxDB memory overhead is only 23.9% of the original InfluxDB system.