skip to main content
10.1145/3229710.3229715acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicppConference Proceedingsconference-collections
research-article

Run-Length Base-Delta Encoding for High-Speed Compression

Published: 13 August 2018 Publication History

Abstract

In modern supercomputers, nodes are connected by networking hardware capable of up to 40 Gb/s. Data compression could allow for even higher effective bandwidth. However, data compression for such systems requires a unique tradeoff between the compression rate delivered by the compressing scheme and the speed of compression/decompression. While traditional software compression techniques may deliver high compression rates, they cannot maintain the high compression/decompression throughput that is needed. We present Run-Length Base-Delta (RLBD) encoding, a software compression format and algorithm that delivers a highspeed compression/decompression that is suitable for data transfers at up to 40GbE. RLBD can be implemented in CPUs or in parallel accelerator devices to improve network throughput by up to 57% while transmitting data from real datasets.

References

[1]
Jyrki Alakuijala and Zoltan Szabadka. 2016. Brotli compressed data format. Technical Report. Google Inc. https://tools.ietf.org/html/rfc7932
[2]
United States Department of Commerce Bureau of the Census. 2006. Census of Population and Housing, 1990 {United States}: Summary Tape File 3A, Record Sequence Example File. (2006).
[3]
Yann Collet. 2013. LZ4 lossless compression algorithm. (2013). https://code.google.com/lz4
[4]
Stephen Curial, Peng Zhao, Jose Nelson Amaral, Yaoqing Gao, Shimin Cui, Raul Silvera, and Roch Archambault. 2008. MPADS: memory-pooling-assisted data splitting. In 7th international symposium on Memory management. ACM, Tucson, AZ, USA, 101--110.
[5]
David Goldberg. 1991. What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys (CSUR) 23, 1 (1991), 5--48.
[6]
Google. 2011. Snappy: A fast compressor/decompressor. (2011). https://gir.hub.com/google/snappy
[7]
Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. 2005. Result analysis of the NIPS 2003 feature selection challenge. In Advances in neural information processing systems. 545--552.
[8]
Rolf Hempel. 1994. The MPI standard for message passing. In International Conference on High-Performance Computing and Networking. Springer, Munich, Germany, 247--252.
[9]
Oak Ridge National Laboratory. 2018. Oak Ridge Leadership Computing Facility - Summit. (2018). https://www.olcf.ornl.gov/summit/
[10]
Rastislav Lenhardt and Jyrki Alakuijala. 2012. Gipfeli-high speed compression algorithm. In Data Compression Conference (DCC). Los Alamitos, CA, USA, 109--118.
[11]
M. Lichman. 2013. UCI Machine Learning Repository. (2013). http://archive.ics.uci.edu/ml
[12]
NVIDIA. 2017. NVIDIA Tesla V100 Datasheet. (2017). http://www.nvidia.com/content/PDF/Volta-Datasheet.pdf
[13]
Gennady Pekhimenko, Vivek Seshadri, Onur Mutlu, Phillip B Gibbons, Michael A Kozuch, and Todd C Mowry. 2012. Base-delta-immediate compression: Practical data compression for on-chip caches. In Parallel architectures and compilation techniques (PACT). ACM, 377--388.
[14]
Krishnaprasad Shastry, Avinash Pandey, Ashutosh Agrawal, and Ravi Sarveswara. 2016. Compression Acceleration Using GPGPU. In High Performance Computing Workshop (HiPC). Hyderabad, India, 70--78.
[15]
Evangelia Sitaridi, Rene Mueller, Tim Kaldewey, Guy Lohman, and Kenneth A Ross. 2016. Massively-parallel lossless data decompression. In 45th International Conference on Parallel Processing (ICPP). Philadelphia, PA, USA, 242--247.
[16]
Przemyslaw Skibinski. 2017. Lzbench. (2017). https://github.com/inikep/lzbench
[17]
Manuel Ujaldón. 2016. CUDA Achievements and GPU Challenges Ahead. In International Conference on Articulated Motion and Deformable Objects. Springer, 207--217.
[18]
Eduardo Velloso, Andreas Bulling, Hans Gellersen, Wallace Ugulino, and Hugo Fuks. 2013. Qualitative activity recognition of weight lifting exercises. In 4th Augmented Human International Conference. Stuttgart, Germany, 116--123.
[19]
Jia Zhan, Matt Poremba, Yi Xu, and Yuan Xie. 2014. NoΔ: Leveraging delta compression for end-to-end memory access in NoC based multicores. In Asia and South Pacific Design Automation Conference (ASP-DAC). Singapore, 586--591.

Cited By

View all
  • (2022)A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency DistributionsApplied Sciences10.3390/app1204204412:4(2044)Online publication date: 16-Feb-2022
  • (2022)IoT streamed data handling model using delta encodingInternational Journal of Communication Systems10.1002/dac.524335:13Online publication date: 3-Jun-2022
  • (2020)An Enhanced Data Compression Algorithm2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE)10.1109/ic-ETITE47903.2020.223(1-4)Online publication date: Feb-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICPP Workshops '18: Workshop Proceedings of the 47th International Conference on Parallel Processing
August 2018
409 pages
ISBN:9781450365239
DOI:10.1145/3229710
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

  • University of Oregon: University of Oregon

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 August 2018

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICPP '18 Comp

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)24
  • Downloads (Last 6 weeks)2
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency DistributionsApplied Sciences10.3390/app1204204412:4(2044)Online publication date: 16-Feb-2022
  • (2022)IoT streamed data handling model using delta encodingInternational Journal of Communication Systems10.1002/dac.524335:13Online publication date: 3-Jun-2022
  • (2020)An Enhanced Data Compression Algorithm2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE)10.1109/ic-ETITE47903.2020.223(1-4)Online publication date: Feb-2020
  • (2020)Improved data transfer efficiency for scale‐out heterogeneous workloads using on‐the‐fly I/O link compressionConcurrency and Computation: Practice and Experience10.1002/cpe.610135:11Online publication date: Dec-2020
  • (2019)Evaluation of Encoding Schemas for Optimization of Bit-Level Run-Length Encoding Within Lossless Compression of Binary Images2019 IEEE 23rd International Conference on Intelligent Engineering Systems (INES)10.1109/INES46365.2019.9109528(000075-000080)Online publication date: Apr-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media