Skip to main content
SpringerLink
Log in

Cuttle: Enabling Cross-Column Compression in Distributed Column Stores

  • Conference paper
  • First Online:
Web and Big Data (APWeb-WAIM 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10367))

Abstract

We observe that, in real-world distributed data warehouse systems, data columns from different sources often exhibit redundancy. Even though these systems can employ both general and column-oriented compression schemes to reduce the data storage pressure, such cross-column redundancy (CCR) is not recognized or exploited effectively. Therefore, we propose Cuttle, a column storage system that enables cross-column compression to reduce CCR. Specifically, we identify three kinds of CCR and develop a referential transformation encoding (RTE) scheme to compress multiple columns of data with CCR. Furthermore, we address the CCR selection problem and propose a greedy algorithm to generate cross-column compression schemes. Our experiments on real-world datasets show that Cuttle can further reduce data size by half after applying both the column-oriented and general compression schemes, and that the query processing performance with Cuttle is improved by \(20\%\) without any change to the application programs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://en.unionpay.com/.

  2. 2.

    https://www.transtats.bts.gov/.

  3. 3.

    https://www.kaggle.com/maxhorowitz/nflplaybyplay2015.

References

  1. Gzip. http://man.openbsd.org/OpenBSD-current/man1/gzip

  2. Lz. https://www.lzop.org

  3. Vertica. https://www.vertica.com/

  4. Zlib. http://www.zlib.net/

  5. Abadi, D., Madden, S., Ferreira, M.: Integrating compression and execution in column-oriented database systems. In: Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, pp. 671–682. ACM (2006)

    Google Scholar 

  6. Idreos, S., Groffen, F., Nes, N., Manegold, S., Mullender, S., Kersten, M., et al.: MonetDB: two decades of research in column-oriented database architectures. Bull. IEEE Comput. Soc. Tech. Committee Data Eng. 35(1), 40–45 (2012)

    Google Scholar 

  7. Roth, M.A., Van Horn, S.J.: Database compression. SIGMOD Rec. 22, 31–39 (1993)

    Article  Google Scholar 

  8. Stonebraker, M., Abadi, D.J., Batkin, A., Chen, X., Cherniack, M., Ferreira, M., Lau, E., Lin, A., Madden, S., O’Neil, E., O’Neil, P., Rasin, A., Tran, N., Zdonik, S.: C-store: a column-oriented DBMS. In: VLDB, pp. 553–564 (2005)

    Google Scholar 

  9. Thusoo, A., Sarma, J.S., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., Murthy, R.: Hive: a warehousing solution over a map-reduce framework. VLDB 2, 1626–1629 (2009)

    Google Scholar 

  10. Westmann, T., Kossmann, D., Helmer, S., Moerkotte, G.: The implementation and performance of compressed databases. SIGMOD Rec. 29, 55–67 (2000)

    Article  Google Scholar 

Download references

Acknowledgment

This research is supported in part by the National Key Basic Research and Development Program of China (973) Grant 2014CB340303.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong

    Hao Liu, Haoyu Tan & Qiong Luo

  2. Huazhong University of Science and Technology, Wuhan, China

    Jiang Xiao

  3. Deepera Inc., Ocean Coast City, Shenzhen, China

    Xianjun Guo

  4. University of Macau, Zhuhai, China

    Lionel M. Ni

Authors
  1. Hao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiang Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xianjun Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haoyu Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lionel M. Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiang Xiao .

Editor information

Editors and Affiliations

  1. Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China

    Lei Chen

  2. Computer Science, Aarhus University, Aarhus N, Denmark

    Christian S. Jensen

  3. Computer Science, University of Southern California, Los Angeles, California, USA

    Cyrus Shahabi

  4. Northeastern University , Shenyang, China

    Xiaochun Yang

  5. Kent State University, Kent, Ohio, USA

    Xiang Lian

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Liu, H., Xiao, J., Guo, X., Tan, H., Luo, Q., Ni, L.M. (2017). Cuttle: Enabling Cross-Column Compression in Distributed Column Stores. In: Chen, L., Jensen, C., Shahabi, C., Yang, X., Lian, X. (eds) Web and Big Data. APWeb-WAIM 2017. Lecture Notes in Computer Science(), vol 10367. Springer, Cham. https://doi.org/10.1007/978-3-319-63564-4_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-63564-4_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-63563-7

  • Online ISBN: 978-3-319-63564-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation