research-article

Brotli: A General-Purpose Data Compressor

Authors:

Jyrki Alakuijala,

Andrea Farruggia,

Paolo Ferragina,

Eugene Kliuchnikov,

Zoltan Szabadka,

Lode VandevenneAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 37, Issue 1

Article No.: 4, Pages 1 - 30

https://doi.org/10.1145/3231935

Published: 06 December 2018 Publication History

Abstract

Brotli is an open source general-purpose data compressor introduced by Google in late 2013 and now adopted in most known browsers and Web servers. It is publicly available on GitHub and its data format was submitted as RFC 7932 in July 2016. Brotli is based on the Lempel-Ziv compression scheme and planned as a generic replacement of Gzip and ZLib. The main goal in its design was to compress data on the Internet, which meant optimizing the resources used at decoding time, while achieving maximal compression density.

This article is intended to provide the first thorough, systematic description of the Brotli format as well as a detailed computational and experimental analysis of the main algorithmic blocks underlying the current encoder implementation, together with a comparison against compressors of different families constituting the state-of-the-art either in practice or in theory. This treatment will allow us to raise a set of new algorithmic and software engineering problems that deserve further attention from the scientific community.

References

[1]

Jyrki Alakuijala and Zoltan Szabadka. 2016. Brotli compressed data format (RFC 7932). IETF: Internet Engineering Task Force (July 2016).

[2]

Brian Allen and J. Ian Munro. 1978. Self-organizing binary search trees. J. ACM 25, 4 (1978), 526--535.

Digital Library

[3]

Michael A. Bender and Martin Farach-Colton. 2004. The level ancestor problem simplified. Theor. Comput. Sci. 321, 1 (2004), 5--12.

Digital Library

[4]

Michael Burrows and David J. Wheeler. 1994. A Block-Sorting Lossless Data Compression Algorithm. Technical Report. Digital Corp.

[5]

Yann Collet. 2011. The Compressor LZ4. Retrieved from https://en.wikipedia.org/wiki/LZ4_(compression_algorithm).

[6]

Yann Collet. 2016. The Compressor ZStd. Retrieved from http://www.zstd.net.

[7]

Lasse Collin. 2010. XZ Utils. Retrieved from https://tukaani.org/xz/.

[8]

Jarek Duda, Khalid Tahboub, Neeraj J. Gadgil, and Edward J. Delp. 2015. The use of asymmetric numeral systems as an accurate replacement for Huffman coding. In Proceedings of the 2015 Picture Coding Symposium (PCS’15). 65--69.

[9]

Andrea Farruggia, Paolo Ferragina, Antonio Frangioni, and Rossano Venturini. 2014. Bicriteria data compression. In Proceedings of the 25th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’14). ACM-SIAM, 1582--1595.

Digital Library

[10]

Andrea Farruggia, Paolo Ferragina, and Rossano Venturini. 2014. Bicriteria data compression: Efficient and usable. In Proceedings of the European Symposium on Algorithms (ESA’14). Lecture Notes in Computer Science, Vol. 8737. Springer, 406--417.

[11]

Paolo Ferragina, Raffaele Giancarlo, Giovanni Manzini, and Marinella Sciortino. 2005. Boosting textual compression in optimal linear time. J. ACM 52, 4 (2005), 688--713.

Digital Library

[12]

Paolo Ferragina, Igor Nitto, and Rossano Venturini. 2011. On optimally partitioning a text to improve its compression. Algorithmica 61, 1 (2011), 51--74.

Digital Library

[13]

Paolo Ferragina, Igor Nitto, and Rossano Venturini. 2013. On the bit-complexity of Lempel-Ziv compression. SIAM J. Comput. 42, 4 (2013), 1521--1541.

[14]

Rich Geldreich. 2010. The Compressor LZHAM. Retrieved from https://github.com/richgel999/lzham_codec.

[15]

Eran Halperin and Richard M. Karp. 2005. The minimum-entropy set cover problem. Theor. Comput. Sci. 348, 2--3 (2005), 240--250.

Digital Library

[16]

Apple Inc. 2015. The Compressor LZFSE. Retrieved from https://github.com/lzfse/lzfse.

[17]

Sanjay Jeff Dean, Steinar Ghemawat, and H. Gunderson. 2011. The Compressor Snappy. Retrieved from https://en.wikipedia.org/wiki/Snappy_(compression).

[18]

Shmuel T. Klein. 2000. Improving static compression schemes by alphabet extension. In Proceedings of the 11th Combinatorial Pattern Matching Conference (CPM’00). 210--221.

Digital Library

[19]

Alistair Moffat. 1990. Implementing the PPM data compression scheme. IEEE Trans. Commun. 38, 11 (1990), 1917--1921.

[20]

Igor Pavlov. 1998. The Algorithm: Lempel-Ziv-Markov Chain. Retrieved from https://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm.

[21]

David Salomon. 2007. Data Compression: The Complete Reference (4th ed.). Springer-Verlag, Berlin.

Digital Library

[22]

E. J. Schuegraf and H. S. Heaps. 1974. A comparison of algorithms for data base compression by use of fragments as language elements. Inf. Stor. Retriev. 10, 9--10 (1974), 309--319.

[23]

Raimund Seidel and Cecilia R. Aragon. 1996. Randomized search trees. Algorithmica 16, 4/5 (1996), 464--497.

[24]

Daniel Dominic Sleator and Robert Endre Tarjan. 1985. Self-adjusting binary search trees. J. ACM 32, 3 (1985), 652--686.

Digital Library

[25]

Andrew Turpin and Alistair Moffat. 2000. Housekeeping for prefix coding. IEEE Trans. Commun. 48, 4 (2000), 622--628.

[26]

Lode Vandevenne and Jyrki Alakuijala. 2013. Zopfli. Retrieved from https://github.com/google/zopfli.

[27]

I. H. Witten, A. Moffat, and T. C. Bell. 1999. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann Publishers.

Digital Library

[28]

Jacob Ziv and Abraham Lempel. 1977. A universal algorithm for sequential data compression. IEEE Trans. Inf. Theory 23, 3 (1977), 337--343.

Digital Library

[29]

Jacob Ziv and Abraham Lempel. 1978. Compression of individual sequences via variable-rate coding. IEEE Trans. Inf. Theory 24, 5 (1978), 530--536.

Digital Library

Cited By

Zhang LLi RRen CDi SLiu JHuang JUnderwood RGrosset PTao DLiang XGuo HCappello FZhao K(2025)LCP: Enhancing Scientific Data Management with Lossy Compression for ParticlesProceedings of the ACM on Management of Data10.1145/37097003:1(1-27)Online publication date: 11-Feb-2025
https://dl.acm.org/doi/10.1145/3709700
Luu Van Sang Vu Hoang Minh Tran Binh Minh Nguyen Van Trung Nguyen Anh Tuan Dang Duc Trinh (2024)A hybrid terrain data compression method in unity for deployment on resource-limited devicesJournal of Military Science and Technology10.54939/1859-1043.j.mst.94.2024.130-138(130-138)Online publication date: 22-Apr-2024
https://doi.org/10.54939/1859-1043.j.mst.94.2024.130-138
Atasoy FAkkaya AKocakır NKarademir Ö(2024)Performance Analysis of Compression Algorithms on Matrix Data: Data Transfer Optimization in Microservices ArchitecturesTürk Doğa ve Fen Dergisi10.46810/tdfd.1425959(49-60)Online publication date: 1-Oct-2024
https://doi.org/10.46810/tdfd.1425959
Show More Cited By

Index Terms

Brotli: A General-Purpose Data Compressor

Recommendations

LOCO-I: a low complexity, context-based, lossless image compression algorithm
DCC '96: Proceedings of the Conference on Data Compression

LOCO-I (low complexity lossless compression for images) is a novel lossless compression algorithm for continuous-tone images which combines the simplicity of Huffman coding with the compression potential of context models, thus "enjoying the best of ...
Wavelet transform and bit-plane encoding
ICIP '95: Proceedings of the 1995 International Conference on Image Processing (Vol. 1)-Volume 1 - Volume 1

In this work a new approach for wavelet transform (WT) based image compression is presented. Employing a simple region representation coding scheme previously used with bi-level facsimile pictures, the wavelet transform coefficients are first quantized ...
Lazy Lempel-Ziv Factorization Algorithms
Special Issue SEA 2014, Regular Papers and Special Issue ALENEX 2013

For decades the Lempel-Ziv (LZ77) factorization has been a cornerstone of data compression and string processing algorithms, and uses for it are still being uncovered. For example, LZ77 is central to several recent text indexing data structures designed ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 37, Issue 1

January 2019

435 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/3289475

Editor:
Maarten de Rijke
University of Amsterdam, The Netherlands

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 December 2018

Accepted: 01 June 2018

Revised: 01 March 2018

Received: 01 September 2017

Published in TOIS Volume 37, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Google Faculty Research Award

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
830
Total Downloads

Downloads (Last 12 months)125
Downloads (Last 6 weeks)12

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang LLi RRen CDi SLiu JHuang JUnderwood RGrosset PTao DLiang XGuo HCappello FZhao K(2025)LCP: Enhancing Scientific Data Management with Lossy Compression for ParticlesProceedings of the ACM on Management of Data10.1145/37097003:1(1-27)Online publication date: 11-Feb-2025
https://dl.acm.org/doi/10.1145/3709700
Luu Van Sang Vu Hoang Minh Tran Binh Minh Nguyen Van Trung Nguyen Anh Tuan Dang Duc Trinh (2024)A hybrid terrain data compression method in unity for deployment on resource-limited devicesJournal of Military Science and Technology10.54939/1859-1043.j.mst.94.2024.130-138(130-138)Online publication date: 22-Apr-2024
https://doi.org/10.54939/1859-1043.j.mst.94.2024.130-138
Atasoy FAkkaya AKocakır NKarademir Ö(2024)Performance Analysis of Compression Algorithms on Matrix Data: Data Transfer Optimization in Microservices ArchitecturesTürk Doğa ve Fen Dergisi10.46810/tdfd.1425959(49-60)Online publication date: 1-Oct-2024
https://doi.org/10.46810/tdfd.1425959
Emvoliadis AVryzas NStamatiadou MVrysis LDimoulas C(2024)Multimodal Environmental Sensing Using AI & IoT Solutions: A Cognitive Sound Analysis PerspectiveSensors10.3390/s2409275524:9(2755)Online publication date: 26-Apr-2024
https://doi.org/10.3390/s24092755
Yao YChen LFang ZGao YJensen CLi T(2024)Camel: Efficient Compression of Floating-Point Time SeriesProceedings of the ACM on Management of Data10.1145/36988022:6(1-26)Online publication date: 20-Dec-2024
https://dl.acm.org/doi/10.1145/3698802
Pápay LPustelnik JRzadca KStrack BStradomski PWołowiec BZasadzinski MSekar VYu MSeneviratne AVeitch D(2024)An exabyte a day: throughput-oriented, large scale, managed data transfers with EffingoProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672262(970-982)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672262
Lin WZhang YDai WLiu HSee JXiong H(2024)Scene Graph Lossless Compression with Adaptive Prediction for Objects and RelationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364950320:7(1-23)Online publication date: 27-Mar-2024
https://dl.acm.org/doi/10.1145/3649503
Lulu Y(2024)Comparative Analysis of Statistical Test Based on Data Compression Methods and Standard Tests for Assessing Randomness of Random Number Generators2024 IEEE International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON)10.1109/SIBIRCON63777.2024.10758518(19-24)Online publication date: 30-Sep-2024
https://doi.org/10.1109/SIBIRCON63777.2024.10758518
Stumpp DAkolkar HGeorge ABenosman R(2024)Flow-Based Visual Stream Compression for Event CamerasIEEE Internet of Things Journal10.1109/JIOT.2024.345042811:24(40229-40243)Online publication date: 15-Dec-2024
https://doi.org/10.1109/JIOT.2024.3450428
Abel ALi YO'Grady RKennelly CGove D(2024)A Profiling-Based Benchmark Suite for Warehouse-Scale Computers2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)10.1109/ISPASS61541.2024.00046(325-327)Online publication date: 5-May-2024
https://doi.org/10.1109/ISPASS61541.2024.00046
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents