Skip to main content
Springer Nature Link
Log in

Multilevel techniques for compression and reduction of scientific data—the univariate case

  • Original Article
  • Published:
Computing and Visualization in Science

Abstract

We present a multilevel technique for the compression and reduction of univariate data and give an optimal complexity algorithm for its implementation. A hierarchical scheme offers the flexibility to produce multiple levels of partial decompression of the data so that each user can work with a reduced representation that requires minimal storage whilst achieving the required level of tolerance. The algorithm is applied to the case of turbulence modelling in which the datasets are traditionally not only extremely large but inherently non-smooth and, as such, rather resistant to compression. We decompress the data for a range of relative errors, carry out the usual analysis procedures for turbulent data, and compare the results of the analysis on the reduced datasets to the results that would be obtained on the full dataset. The results obtained demonstrate the promise of multilevel compression techniques for the reduction of data arising from large scale simulations of complex phenomena such as turbulence modelling.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ainsworth, M., Klasky, S., Whitney, B.: Compression using lossless decimation: analysis and application. SIAM J. Sci. Comput. 39(4), B732–B757 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  2. Austin, W., Ballard, G., Kolda, T. G.: Parallel tensor compression for large-scale scientific data. In: 2016 IEEE international parallel and distributed processing symposium (IPDPS), pp. 912–922, May 2016

  3. Bank, R.E., Dupont, T.F., Yserentant, H.: The hierarchical basis multigrid method. Numer. Math. 52(4), 427–458 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bautista, G., Leonardo, A., Cappello, F.: Improving floating point compression through binary masks. In: 2013 IEEE international conference on big data, pp. 326–331, October 2013

  5. Bornemann, F., Yserentant, H.: A basic norm equivalence for the theory of multilevel methods. Numer. Math. 64(1), 455–476 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  6. Burtscher, M., Hari, M., Annie, Y., Farbod, H.: Real-time synthesis of compression algorithms for scientific data. In: SC ‘16: proceedings of the international conference for high performance computing, networking, storage and analysis, IEEE, pp. 264–275, November 2016

  7. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley Series in Telecommunications, 1st edn. Wiley, New York (1991)

    Book  MATH  Google Scholar 

  8. Dahmen, W., Kunoth, A.: Multilevel preconditioning. Numer. Math. 63(3), 315–344 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  9. Daubechies, I.: The wavelet transform, time-frequency localization and signal analysis. IEEE Trans. Inf. Theory 36(5), 961–1005 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  10. Di, S., Cappello, F.: Fast error-bounded lossy HPC data compression with SZ. In: 2016 IEEE 30th international parallel and distributed processing symposium, IEEE, Chicago, IL, USA, pp. 730–739, May 2016

  11. Donoho, D.L., Vetterli, M., DeVore, R.A., Daubechies, I.: Data compression and harmonic analysis. IEEE Trans. Inf. Theory 44(6), 2435–2476 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  12. Edmunds, D.E., Triebel, H.: Function Spaces, Entropy Numbers, Differential Operators, 1st edn. Cambridge University Press, Cambridge (1996)

    Book  MATH  Google Scholar 

  13. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. The Johns Hopkins University Press, Baltimore (1996)

    MATH  Google Scholar 

  14. Grgic, S., Kers, K., Grgic, M.: Image compression using wavelets. In: Proceedings of the IEEE international symposium on industrial electronics, 1999. ISIE ‘99, vol. 1, pp. 99–104 (1999)

  15. Griebel, M., Oswald, P.: Stable splittings of Hilbert spaces of functions of infinitely many variables. J. Complex. 41, 126–151 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  16. Ibarria, L., Lindstrom, P., Rossignac, J., Szymczak, A.: Out-of-core compression and decompression of large n-dimensional scalar fields. Comput. Graph. Forum 22(3), 343–348 (2003)

    Article  Google Scholar 

  17. Johns Hopkins Turbulence Databases. Forced isotropic turbulence dataset description, October 2017. Last update: 10/19/2017 5:55:14 PM. Accessed 01 Feb 2018

  18. Kolmogorov, A.: The local structure of turbulence in incompressible viscous fluid for very large Reynolds’ numbers. Akademiia Nauk SSSR Doklady 30, 301–305 (1941)

    MathSciNet  Google Scholar 

  19. Lakshminarasimhan, S., Shah, N., Ethier, S., Klasky, S., Latham, R., Ross, R., Samatova, N. F.: Compressing the incompressible with ISABELA: in-situ reduction of spatio-temporal data. In: Emmanuel J., Raymond N., Jean R. (eds) Euro-Par 2011: Parallel Processing Workshops, Lecture Notes in Computer Science, Bordeaux, France, Springer, Berlin, Heidelberg, vol. 6852, pp. 366–379, August 2011

  20. Li, Y., Perlman, E., Wan, M., Yang, Y., Meneveau, C., Burns, R., Chen, S., Szalay, A., Eyink, G.: A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. J. Turbul. 9, N31 (2008)

    Article  MATH  Google Scholar 

  21. Lindstrom, P.: Fixed-rate compressed floating-point arrays. IEEE Trans. Vis. Comput. Graph. 20(12), 2674–2683 (2014)

    Article  Google Scholar 

  22. Lindstrom, P., Isenburg, M.: Fast and efficient compression of floating-point data. IEEE Trans. Vis. Comput. Graph. 12(5), 1245–1250 (2006)

    Article  Google Scholar 

  23. Marcellin, M. W., Gormish, M. J., Bilgin, A., Boliek, M. P.: An overview of JPEG-2000. In: Proceedings DCC 2000. Data compression conference, pp. 523–541 (2000)

  24. Oswald, P.: Multilevel Finite Element Approximation. Theory and Applications. Teubner Skripten zur Numerik. B. G. Teubner, Stuttgart (1994)

    Book  MATH  Google Scholar 

  25. Perlman, E., Burns, R., Li, Y., Meneveau, C.: Data exploration of turbulence simulations using a database cluster. In: Proceedings of the 2007 ACM/IEEE conference on supercomputing, ACM, Reno, NV, USA, vol. 23, November 2007

  26. Salomon, D.: Data Compression: The Complete Reference, 4th edn. Springer, London (2007)

    MATH  Google Scholar 

  27. Schendel, E. R., Jin, Y., Shah, N., Chen, J., Chang, C. S., Ku, S.-H., Ethier, S., Klasky, S., Latham, R., Ross, R., Samatova, N. F.: ISOBAR preconditioner for effective and high-throughput lossless data compression. In: 2012 IEEE 28th international conference on data engineering, pp. 138–149, April 2012

  28. Schneider, K., Farge, M., Pellegrino, G., Rogers, M.M.: Coherent vertex simulation of three-dimensional turbulent mixing layers using orthogonal wavelets. J. Fluid Mech. 534, 39–66 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  29. Shah, N., Schendel, E. R., Lakshminarasimhan, S., Pendse, S. V., Rogers, T., Samatova, N. F.: Improving I/O throughput with PRIMACY: preconditioning ID-mapper for compressing incompressibility. In: 2012 IEEE international conference on cluster computing, pp. 209–219, September 2012

  30. Strengert, M., Magallón, M., Weiskopf, D., Guthe, S., Ertl, T.: Hierarchical visualization and compression of large volume datasets using GPU clusters. EGPGV, pp. 41–48 (2004)

  31. Wallace, G. K.: The JPEG still picture compression standard. IEEE Trans. Consum. Electron. 38(1), xviii–xxxiv (1992)

Download references

Author information

Authors and Affiliations

  1. Division of Applied Mathematics, Brown University, 182 George St., Providence, RI, 02912, USA

    Mark Ainsworth, Ozan Tugluk & Ben Whitney

  2. Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831, USA

    Mark Ainsworth & Scott Klasky

  3. Department of Astronautical Engineering, University of Turkish Aeronautical Association, 06790, Ankara, Turkey

    Ozan Tugluk

Authors
  1. Mark Ainsworth
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ozan Tugluk
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ben Whitney
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Scott Klasky
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Mark Ainsworth.

Additional information

This work is dedicated to Prof. Ulrich Langer on the occasion of his sixtieth birthday.

This research was supported in part by the Exascale Computing Project (17-SC-20-SC) of the U.S. Department of Energy; the Advanced Scientific Research Office (ASCR) at the Department of Energy, under contract DE-AC02-06CH11357; the DOE Storage Systems and Input/Output for Extreme Scale Science project, announcement number LAB 15-1338; and DOE and UT–Battelle, LLC, Contract Number DE-AC05-00OR22725.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ainsworth, M., Tugluk, O., Whitney, B. et al. Multilevel techniques for compression and reduction of scientific data—the univariate case. Comput. Visual Sci. 19, 65–76 (2018). https://doi.org/10.1007/s00791-018-00303-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00791-018-00303-9

Keywords