skip to main content
10.1145/2594291.2594302acmconferencesArticle/Chapter ViewAbstractPublication PagespldiConference Proceedingsconference-collections
research-article

Stochastic optimization of floating-point programs with tunable precision

Published: 09 June 2014 Publication History

Abstract

The aggressive optimization of floating-point computations is an important problem in high-performance computing. Unfortunately, floating-point instruction sets have complicated semantics that often force compilers to preserve programs as written. We present a method that treats floating-point optimization as a stochastic search problem. We demonstrate the ability to generate reduced precision implementations of Intel's handwritten C numeric library which are up to 6 times faster than the original code, and achieve end-to-end speedups of over 30% on a direct numeric simulation and a ray tracer by optimizing kernels that can tolerate a loss of precision while still remaining correct. Because these optimizations are mostly not amenable to formal verification using the current state of the art, we present a stochastic search technique for characterizing maximum error. The technique comes with an asymptotic guarantee and provides strong evidence of correctness.

References

[1]
C. Andrieu, N. de Freitas, A. Doucet, and M. I. Jordan. An introduction to MCMC for machine learning. Machine Learning, 50(1--2):5--43, 2003.
[2]
W. Baek and T. M. Chilimbi. Green: A framework for supporting energy-conscious programming using controlled approximation. In PLDI, pages 198--209, 2010.
[3]
F. Benz, A. Hildebrandt, and S. Hack. A dynamic program analysis to find floating-point accuracy problems. In PLDI, pages 453--462, 2012.
[4]
S. Chaudhuri, S. Gulwani, and R. Lublinerman. Continuity and robustness of programs. Commun. ACM, 55(8):107--115, 2012.
[5]
W.-F. Chiang, G. Gopalakrishnan, Z. Rakamaric, and A. Solovyev. Efficient search for inputs causing high floating-point errors. PPoPP '14, pages 43--52, 2014.
[6]
P. Collingbourne, C. Cadar, and P. H. J. Kelly. Symbolic crosschecking of floating-point and SIMD code. In EuroSys, pages 315--328, 2011.
[7]
C. Curtsinger and E. D. Berger. Stabilizer: Statistically sound performance evaluation. In ASPLOS, pages 219--228, 2013.
[8]
E. Darulova and V. Kuncak. Sound compilation of reals. In POPL, pages 235--248, 2014.
[9]
F. de Dinechin, C. Q. Lauter, and G. Melquiond. Certifying the floating-point implementation of an elementary function using GAPPA. IEEE Trans. Computers, 60(2):242--253, 2011.
[10]
L. M. de Moura, N. Bjørner, and C. M. Wintersteiger. Z3. http://z3.codeplex.com/.
[11]
D. Delmas, E. Goubault, S. Putot, J. Souyris, K. Tekkal, and F. Védrine. Towards an industrial use of FLUCTUAT on safety-critical avionics software. In FMICS, pages 53--69, 2009.
[12]
J. Geweke. Priors for macroeconomic time series and their application. Technical report, Federal Reserve Bank of Minneapolis, 1992.
[13]
D. Goldberg. What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1):5--48, 1991.
[14]
S. Gulwani, S. Jha, A. Tiwari, and R. Venkatesan. Synthesis of loop-free programs. In PLDI, pages 62--73, 2011.
[15]
L. Haller, A. Griggio, M. Brain, and D. Kroening. Deciding floating-point logic with systematic abstraction. In FMCAD, pages 131--140, 2012.
[16]
P. S. Heckbert, editor. Graphics gems IV. Academic Press Professional, Inc., San Diego, CA, USA, 1994.
[17]
F. Ivancic, M. K. Ganai, S. Sankaranarayanan, and A. Gupta. Numerical stability analysis of floating-point computations using software model checking. In MEMOCODE, pages 49--58, 2010.
[18]
M. Jha and S. Raskhodnikova. Testing and reconstruction of lipschitz functions with applications to data privacy. Electronic Colloquium on Computational Complexity (ECCC), pages 11--57, 2011.
[19]
F. Jzquel and J. M. Chesneaux. CADNA: A library for estimating round-off error propagation. Computer Physics Communications, 178 (12):933--955, 2008.
[20]
K. Lakhotia, N. Tillmann, M. Harman, and J. de Halleux. FloPSy - Search-based floating-point constraint solving for symbolic execution. In ICTSS, pages 142--157, 2010.
[21]
M. O. Lam, J. K. Hollingsworth, B. R. de Supinski, and M. P. Legendre. Automatically adapting programs for mixed-precision floating-point computation. ICS '13, pages 369--378, 2013.
[22]
S. Misailovic, D. M. Roy, and M. C. Rinard. Probabilistically accurate program transformations. In SAS, pages 316--333, 2011.
[23]
G. C. Necula and S. Gulwani. Randomized algorithms for program analysis and verification. In CAV, 2005.
[24]
A. Podgurski. Reliability, sampling, and algorithmic randomness. TAV4, pages 11--20, New York, NY, USA, 1991. ACM.
[25]
R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2008. http://www.R-project.org.
[26]
C. Rubio-González, C. Nguyen, H. D. Nguyen, J. Demmel, W. Kahan, K. Sen, D. H. Bailey, C. Iancu, and D. Hough. Precimonious: Tuning assistant for floating-point precision. In SC, 2013.
[27]
P. Rümmer and T. Wahl. An SMT-LIB theory of binary floating-point arithmetic. In SMT, 2010.
[28]
S. Sankaranarayanan, R. M. Chang, G. Jiang, and F. Ivancic. State space exploration using feedback constraint generation and Monte Carlo sampling. In ESEC/SIGSOFT FSE, pages 321--330, 2007.
[29]
E. Schkufza, R. Sharma, and A. Aiken. Stochastic superoptimization. In ASPLOS, pages 305--316, 2013.
[30]
R. Sharma, E. Schkufza, B. R. Churchill, and A. Aiken. Data-driven equivalence checking. In OOPSLA, pages 391--406, 2013.
[31]
S. Sidiroglou-Douskos, S. Misailovic, H. Hoffmann, and M. C. Rinard. Managing performance vs. accuracy trade-offs with loop perforation. In SIGSOFT FSE, pages 124--134, 2011.
[32]
A. Solar-Lezama, L. Tancau, R. Bodík, S. A. Seshia, and V. A. Saraswat. Combinatorial sketching for finite programs. In ASPLOS, pages 404--415, 2006.
[33]
D. Stott Parker, B. Pierce, and P. R. Eggert. Monte Carlo arithmetic: how to gamble with floating point and win. Computing in Science & Engineering, 2(4):58--68, 2000.
[34]
E. Tang, E. T. Barr, X. Li, and Z. Su. Perturbing numerical calculations for statistical analysis of floating-point program (in)stability. In ISSTA, pages 131--142, 2010.
[35]
C. W. Ueberhuber. Numerical computation: Methods, software and analysis. Springer-Verlag, 1997.
[36]
Z. A. Zhu, S. Misailovic, J. A. Kelner, and M. C. Rinard. Randomized accuracy-aware program transformations for efficient approximate computations. In POPL, pages 441--454, 2012.

Cited By

View all
  • (2024)Numerical Fuzz: A Type System for Rounding Error AnalysisProceedings of the ACM on Programming Languages10.1145/36564568:PLDI(1954-1978)Online publication date: 20-Jun-2024
  • (2023)Inductive Program Synthesis Guided by Observational Program SimilarityProceedings of the ACM on Programming Languages10.1145/36228307:OOPSLA2(912-940)Online publication date: 16-Oct-2023
  • (2023)Comparative Synthesis: Learning Near-Optimal Network Designs by QueryProceedings of the ACM on Programming Languages10.1145/35711977:POPL(91-120)Online publication date: 11-Jan-2023
  • Show More Cited By

Index Terms

  1. Stochastic optimization of floating-point programs with tunable precision

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    PLDI '14: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation
    June 2014
    619 pages
    ISBN:9781450327848
    DOI:10.1145/2594291
    • cover image ACM SIGPLAN Notices
      ACM SIGPLAN Notices  Volume 49, Issue 6
      PLDI '14
      June 2014
      598 pages
      ISSN:0362-1340
      EISSN:1558-1160
      DOI:10.1145/2666356
      • Editor:
      • Andy Gill
      Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 09 June 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. 64-bit
    2. MCMC
    3. Markov Chain Monte Carlo
    4. SMT
    5. binary
    6. floating-point
    7. precision
    8. stochastic search
    9. x86
    10. x86-64

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    PLDI '14
    Sponsor:

    Acceptance Rates

    PLDI '14 Paper Acceptance Rate 52 of 287 submissions, 18%;
    Overall Acceptance Rate 406 of 2,067 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)67
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Numerical Fuzz: A Type System for Rounding Error AnalysisProceedings of the ACM on Programming Languages10.1145/36564568:PLDI(1954-1978)Online publication date: 20-Jun-2024
    • (2023)Inductive Program Synthesis Guided by Observational Program SimilarityProceedings of the ACM on Programming Languages10.1145/36228307:OOPSLA2(912-940)Online publication date: 16-Oct-2023
    • (2023)Comparative Synthesis: Learning Near-Optimal Network Designs by QueryProceedings of the ACM on Programming Languages10.1145/35711977:POPL(91-120)Online publication date: 11-Jan-2023
    • (2023)Deep Reinforcement Learning Guided Decision Tree Learning For Program Synthesis2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00112(925-932)Online publication date: Mar-2023
    • (2023)CryptOpt: Automatic Optimization of Straightline Code2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion58688.2023.00042(141-145)Online publication date: May-2023
    • (2022)A Compiler for Sound Floating-Point Computations using Affine Arithmetic2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)10.1109/CGO53902.2022.9741286(66-78)Online publication date: 2-Apr-2022
    • (2022)Detecting High Floating-Point Errors via Ranking Analysis2022 29th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC57359.2022.00052(397-406)Online publication date: Dec-2022
    • (2022)Accuracy-Aware CompilersApproximate Computing Techniques10.1007/978-3-030-94705-7_7(177-214)Online publication date: 3-Jan-2022
    • (2021)Rewrite rule inference using equality saturationProceedings of the ACM on Programming Languages10.1145/34854965:OOPSLA(1-28)Online publication date: 15-Oct-2021
    • (2021)Regime Inference for Sound Floating-Point OptimizationsACM Transactions on Embedded Computing Systems10.1145/347701220:5s(1-23)Online publication date: 17-Sep-2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media