Scientific Data Lossless Compression Using Fast Neural Network

Abstract

Scientific computing generates huge loads of data from complex simulations, usually takes several TB, general compression methods can not have good performance on these data. Neural networks have the potential to extend data compression algorithms beyond the character level(n-gram model) currently in use, but have usually been avoided because they are too slow to be practical. We present a lossless compression method using fast neural network based on Maximum Entropy and arithmetic coder to succeed in the job. The compressor is a bit-level predictive arithmetic encoder using a 2 layer fast neural network to predict the probability distribution. In the training phase, an improved adaptive variable learning rate is optimized for fast convergence training. The proposed compressor produces better compression than popular compressors(bzip, zzip, lzo, ucl and dflate) on the lared-p data set, also is competitive in time and space for practical application.