skip to main content
10.1145/3425898.3426961acmconferencesArticle/Chapter ViewAbstractPublication PagesgpceConference Proceedingsconference-collections
research-article

Multi-stage programming in the large with staged classes

Published: 16 November 2020 Publication History

Abstract

Multi-stage programming (MSP) holds great promise, allowing the reliable generation of specialized, partially-evaluated code with static type- and scope-safety guarantees. Yet, we argue that MSP has not reached its full potential yet, as it has been traditionally limited to generating expressions, and has lacked principled facilities for generating modular programs and data structures. In that sense, we argue that MSP has been reserved for programming “in the small,” focused on generating efficient kernels of computation on the scale of single function bodies. We present a novel technique called staged classes, which extends MSP with the ability to manipulate class definitions as first-class constructs. This lets programmers use MSP “in the large,” on the level of applications, rather than mere functions. This way, applications can be designed in an abstract and modular way without runtime cost, as staged classes guarantee the removal of all staging-time abstractions, resulting in the generation of efficient specialized modules and data structures. We describe the design of a prototype relational database system in Scala, which uses several stages of runtime compilation to maximize the efficiency of query execution and data storage. We also show that staged classes can be used for defining type- and scope-safe implementations of type providers.

Supplementary Material

Auxiliary Presentation Video (gpce20main-p32-p-video.mp4)
Multi-stage programming (MSP) holds great promise, allowing the reliable generation of specialized, partially-evaluated code with static type- and scope-safety guarantees. Yet, we argue that MSP has not reached its full potential yet, as it has been traditionally limited to generating expressions, and has lacked principled facilities for generating modular programs and data structures. In that sense, we argue that MSP has been reserved for programming “in the small,” focused on generating efficient kernels of computation on the scale of single function bodies. We present a novel technique called staged classes, which extends MSP with the ability to manipulate class definitions as first-class constructs. This lets programmers use MSP “in the large,” on the level of applications, rather than mere functions. This way, applications can be designed in an abstract and modular way without runtime cost, as staged classes guarantee the removal of all staging-time abstractions.
MP4 File (3425898.3426961.mp4)
Presentation Videos

References

[1]
David Abrahams and Aleksey Gurtovoy. 2004. C+ + Template Metaprogramming: Concepts, Tools, and Techniques from Boost and Beyond (C++ in Depth Series). Addison-Wesley Professional.
[2]
Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. Franklin, Ali Ghodsi, and Matei Zaharia. 2015. Spark SQL : Relational Data Processing in Spark. In Proceedings of the ACM SIGMOD International Conference on Management of Data. ACM, New York, NY, USA, 1383-1394.
[3]
Eugene Burmako. 2013. Scala Macros: Let Our Powers Combine!: On How Rich Syntax and Static Types Work with Metaprogramming. In Proceedings of the 4th Workshop on Scala (SCALA '13). ACM, New York, NY, USA, 3 : 1-3 : 10.
[4]
Jacques Carette, Oleg Kiselyov, and Chung-Chieh Shan. 2009. Finally tagless, partially evaluated: Tagless staged interpreters for simpler typed languages. Journal of Functional Programming 19, 05 ( 2009 ), 509-543.
[5]
James Cheney and Ralf Hinze. 2003. First-class phantom types. Technical Report. Cornell University.
[6]
James Cheney, Sam Lindley, and Philip Wadler. 2013. A Practical Theory of Language-integrated Query. In Proceedings of the 18th ACM SIGPLAN International Conference on Functional Programming (ICFP '13). ACM, New York, NY, USA, 403-416.
[7]
Adam Chlipala. 2010. Ur: statically-typed metaprogramming with type-level record computation. In ACM Sigplan Notices, Vol. 45. ACM, 122-133.
[8]
Zachary DeVito, James Hegarty, Alex Aiken, Pat Hanrahan, and Jan Vitek. 2013. Terra: a multi-stage language for high-performance computing. In ACM SIGPLAN Notices, Vol. 48. ACM, 105-116.
[9]
Zachary DeVito, Daniel Ritchie, Matt Fisher, Alex Aiken, and Pat Hanrahan. 2014. First-class runtime generation of high-performance types using exotypes. In Proceedings of the 35th Conference on Programming Language Design and Implementation. ACM, 11.
[10]
Steven E. Ganz, Amr Sabry, and Walid Taha. 2001. Macros as multistage computations: type-safe, generative, binding macros in MacroML. In ACM SIGPLAN Notices, Vol. 36. ACM, 74-85.
[11]
Robert Glück and Jesper Jørgensen. 1997. An automatic program generator for multi-level specialization. Lisp and symbolic computation 10, 2 ( 1997 ), 113-158.
[12]
Google. 2008. Protocol bufers. htps://github.com/protocolbufers/ protobuf. Accessed: 2019-04-06.
[13]
Michael Haidl, Michel Steuwer, Tim Humernbrum, and Sergei Gorlatch. 2016. Multi-stage Programming for GPUs in C++ Using PACXX. In Proceedings of the 9th Annual Workshop on General Purpose Processing Using Graphics Processing Unit (GPGPU '16). ACM, New York, NY, USA, 32-41. htps://doi.org/10.1145/2884045.2884049 event-place: Barcelona, Spain.
[14]
Paul Hudak. 1996. Building domain-specific embedded languages. ACM Computing Surveys (CSUR) 28, 4es ( 1996 ), 196.
[15]
Jun Inoue, Oleg Kiselyov, and Yukiyoshi Kameyama. 2016. Staging Beyond Terms: Prospects and Challenges. In Proceedings of the 2016 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (PEPM '16). ACM, New York, NY, USA, 103-108. htps: //doi.org/10.1145/2847538.2847548
[16]
Manohar Jonnalagedda, Thierry Coppey, Sandro Stucki, Tiark Rompf, and Martin Odersky. 2014. Staged parser combinators for eficient data processing. In Acm Sigplan Notices, Vol. 49. ACM, 637-653.
[17]
Manohar Jonnalagedda and Sandro Stucki. 2015. Fold-based Fusion As a Library: A Generative Programming Pearl. In Proceedings of the 6th ACM SIGPLAN Symposium on Scala (SCALA 2015 ). ACM, Portland, OR, USA, 41-50.
[18]
Nicolai M Josuttis. 2012. The C+ + standard library: a tutorial and reference. Addison-Wesley.
[19]
Vojin Jovanovic, Amir Shaikhha, Sandro Stucki, Vladimir Nikolaev, Christoph Koch, and Martin Odersky. 2014. Yin-Yang: Concealing the Deep Embedding of DSLs (GPCE 2014). ACM, 73-82.
[20]
Yukiyoshi Kameyama, Oleg Kiselyov, and Chung-chieh Shan. 2014. Combinators for Impure Yet Hygienic Code Generation. In Proceedings of the ACM SIGPLAN 2014 Workshop on Partial Evaluation and Program Manipulation (PEPM '14). ACM, New York, NY, USA, 3-14.
[21]
Yukiyoshi Kameyama, Oleg Kiselyov, and Chung-chieh Shan. 2015. Combinators for impure yet hygienic code generation. Science of Computer Programming 112 (part 2) ( 15 Nov. 2015 ), 120-144. htps: //doi.org/10.1016/j.scico. 2015. 08.007
[22]
Ik-Soon Kim, Kwangkeun Yi, and Cristiano Calcagno. 2006. A polymorphic modal type system for lisp-like multi-staged languages. In ACM SIGPLAN Notices, Vol. 41. ACM, 257-268.
[23]
Oleg Kiselyov. 2017. MetaOCaml-an OCaml dialect for multi-stage programming. htps://web.archive.org/web/20170725111517/htp: //okmij.org/ftp/ML/MetaOCaml.html
[24]
Oleg Kiselyov. 2018. Reconciling Abstraction with High Performance: A MetaOCaml Approach. Found. Trends Program. Lang. 5, 1 ( June 2018 ), 1-101. htps://doi.org/10.1561/2500000038
[25]
Oleg Kiselyov, Aggelos Biboudis, Nick Palladinos, and Yannis Smaragdakis. 2017. Stream fusion, to completeness. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages. ACM, 285-299.
[26]
Christoph Koch, Yanif Ahmad, Oliver Kennedy, Milos Nikolic, Andres Nötzli, Daniel Lupei, and Amir Shaikhha. 2014. DBToaster: Higherorder delta processing for dynamic, frequently fresh views. VLDBJ 23, 2 ( 2014 ).
[27]
Eugene Kohlbecker, Daniel P. Friedman, Matthias Felleisen, and Bruce Duba. 1986. Hygienic Macro Expansion. In Proceedings of the 1986 ACM Conference on LISP and Functional Programming (LFP '86). ACM, New York, NY, USA, 151-161.
[28]
Neelakantan R. Krishnaswami and Jeremy Yallop. 2019. A Typed, Algebraic Approach to Parsing (PLDI 2019 ). Association for Computing Machinery, New York, NY, USA, 379-393.
[29]
I. Masliah, M. Baboulin, and J. Falcou. 2016. Meta-programming and Multi-stage Programming for GPGPUs. In 2016 IEEE 10th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSOC). 369-376. htps://doi.org/10.1109/ MCSoC. 2016.49
[30]
Adriaan Moors, Tiark Rompf, Philipp Haller, and Martin Odersky. 2012. Scala-virtualized. In Proceedings of the ACM SIGPLAN 2012 workshop on Partial evaluation and program manipulation. ACM, 117-120.
[31]
Aleksandar Nanevski and Frank Pfenning. 2005. Staged computation with names and necessity. Journal of Functional Programming 15, 6 ( 2005 ), 893-939.
[32]
Thomas Neumann. 2011. Eficiently Compiling Eficient Query Plans for Modern Hardware. PVLDB 4, 9 ( 2011 ), 539-550.
[33]
Gregory Neverov and Paul Roe. 2004. Metaphor: A Multi-stage, ObjectOriented Programming Language. In Generative Programming and Component Engineering (Lecture Notes in Computer Science), Gabor Karsai and Eelco Visser (Eds.). Springer Berlin Heidelberg, 168-185.
[34]
Martin Odersky and Matthias Zenger. 2005. Scalable Component Abstractions. In OOPSLA. San Diego, CA, USA, 41-57.
[35]
Georg Ofenbeck, Tiark Rompf, Alen Stojanov, Martin Odersky, and Markus Püschel. 2013. Spiral in Scala: Towards the Systematic Construction of Generators for Performance Libraries. In Proceedings of the 12th International Conference on Generative Programming: Concepts & Experiences (GPCE '13). ACM, New York, NY, USA, 125-134.
[36]
Junpei Oishi and Yukiyoshi Kameyama. 2017. Staging with Control: Type-safe Multi-stage Programming with Control Operators. In Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE 2017 ). ACM, New York, NY, USA, 29-40. htps://doi.org/10.1145/3136040.3136049 event-place: Vancouver, BC, Canada.
[37]
Bruno C.d.S. Oliveira, Adriaan Moors, and Martin Odersky. 2010. Type Classes As Objects and Implicits. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA '10). ACM, New York, NY, USA, 341-360.
[38]
Lionel Parreaux, Amir Shaikhha, and Christoph E. Koch. 2017. Quoted Staged Rewriting: A Practical Approach to Library-defined Optimizations. In Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE 2017 ). ACM, New York, NY, USA, 131-145. htps://doi.org/10.1145/3136040. 3136043
[39]
Lionel Parreaux, Antoine Voizard, Amir Shaikhha, and Christoph E. Koch. 2017. Unifying Analytic and Statically-typed Quasiquotes. Proc. ACM Program. Lang. 2, POPL (Dec. 2017 ), 13 : 1-13 : 33. htps://doi.org/ 10.1145/3158101
[40]
Tomas Petricek, Gustavo Guerra, and Don Syme. 2016. Types from Data: Making Structured Data First-class Citizens in F#. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '16). ACM, New York, NY, USA, 477-490. htps://doi.org/10.1145/2908080.2908115 event-place: Santa Barbara, CA, USA.
[41]
Tiark Rompf. 2016. Reflections on LMS: exploring front-end alternatives. In Proceedings of the 2016 7th ACM SIGPLAN Symposium on Scala. ACM, 41-50.
[42]
Tiark Rompf and Nada Amin. 2015. Functional pearl: a SQL to C compiler in 500 lines of code. In Proceedings of the 20th ACM SIGPLAN International Conference on Functional Programming, ICFP 2015, Vancouver, BC, Canada, September 1-3, 2015. 2-9.
[43]
Tiark Rompf and Martin Odersky. 2010. Lightweight modular staging: a pragmatic approach to runtime code generation and compiled DSLs. In Generative Programming and Component Engineering. Eindhoven, The Netherlands, 127-136.
[44]
Tiark Rompf, Arvind K. Sujeeth, Nada Amin, Kevin J. Brown, Vojin Jovanovic, HyoukJoong Lee, Manohar Jonnalagedda, Kunle Olukotun, and Martin Odersky. 2013. Optimizing data structures in high-level programs: new directions for extensible compilers based on staging. In POPL. 497-510.
[45]
Adrian Sampson, Kathryn S. McKinley, and Todd Mytkowicz. 2017. Static Stages for Heterogeneous Programming. Proc. ACM Program. Lang. 1, OOPSLA (Oct. 2017 ), 71 : 1-71 : 27. htps://doi.org/10.1145/ 3133895
[46]
Yuhi Sato, Yukiyoshi Kameyama, and Takahisa Watanabe. 2020. Module Generation without Regret. In Proceedings of the 2020 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation ( New Orleans, LA, USA) ( PEPM 2020 ). Association for Computing Machinery, New York, NY, USA, 1-13. htps://doi.org/10.1145/3372884.3373160
[47]
Maximilian Scherr and Shigeru Chiba. 2014. Implicit Staging of EDSL Expressions: A Bridge between Shallow and Deep Embedding. In ECOOP 2014-Object-Oriented Programming: 28th European Conference, Uppsala, Sweden, July 28-August 1, 2014. Proceedings, Richard Jones (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 385-410.
[48]
Maximilian Scherr and Shigeru Chiba. 2015. Almost First-class Language Embedding: Taming Staged Embedded DSLs. In Proceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE 2015 ). ACM, New York, NY, USA, 21-30.
[49]
Amir Shaikhha, Mohammad Dashti, and Christoph Koch. 2018. Push versus Pull-Based Loop Fusion in Query Engines. Journal of Functional Programming 28 ( 2018 ), e10.
[50]
Amir Shaikhha, Vojin Jovanovic, and Christoph E. Koch. 2019. Compiler Generation for Performance-Oriented Embedded DSLs (Short Paper). In Proceedings of the 18th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (Athens, Greece) ( GPCE 2019 ). Association for Computing Machinery, New York, NY, USA, 94-101. htps://doi.org/10.1145/3357765.3359520
[51]
Amir Shaikhha, Yannis Klonatos, and Christoph Koch. 2018. Building Eficient Query Engines in a High-Level Language. ACM Transactions on Database Systems 43, 1, Article 4 ( April 2018 ), 45 pages.
[52]
Amir Shaikhha, Yannis Klonatos, Lionel Parreaux, Lewis Brown, Mohammad Dashti, and Christoph Koch. 2016. How to Architect a Query Compiler. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD '16). ACM, New York, NY, USA, 1907-1922. htps://doi.org/10.1145/2882903.2915244
[53]
Amir Shaikhha and Lionel Parreaux. 2019. Finally, a Polymorphic Linear Algebra Language. In Proceedings of the 33rd European Conference on Object-Oriented Programming (London, United Kingdom) (ECOOP'19).
[54]
Tim Sheard, Zine-el-abidine Benaissa, and Emir Pasalic. 1999. DSL Implementation Using Staging and Monads. In Proceedings of the 2nd Conference on Domain-specific Languages (DSL '99). ACM, New York, NY, USA, 81-94.
[55]
Tim Sheard and Simon Peyton Jones. 2002. Template metaprogramming for Haskell. In Proceedings of the 2002 ACM SIGPLAN workshop on Haskell (Haskell '02). ACM, 1-16.
[56]
Arvind Sujeeth, HyoukJoong Lee, Kevin Brown, Tiark Rompf, Hassan Chafi, Michael Wu, Anand Atreya, Martin Odersky, and Kunle Olukotun. 2011. OptiML: an implicitly parallel domain-specific language for machine learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11). 609-616.
[57]
Kedar Swadi, Walid Taha, Oleg Kiselyov, and Emir Pasalic. 2006. A Monadic Approach for Avoiding Code Duplication When Staging Memoized Functions. In Proceedings of the 2006 ACM SIGPLAN Symposium on Partial Evaluation and Semantics-Based Program Manipulation (Charleston, South Carolina) (PEPM '06). Association for Computing Machinery, New York, NY, USA, 160-169. htps://doi.org/10.1145/ 1111542.1111570
[58]
Don Syme, Keith Battocchi, Kenji Takeda, Donna Malayeri, Jomo Fisher, Jack Hu, Tao Liu, Brian McNamara, Daniel Quirk, Matteo Taveggia, Wonseok Chae, Uladzimir Matsveyeu, and Tomas Petricek. 2012. F# 3. 0-Strongly-Typed Language Support for Internet-Scale Information Sources. (Sept. 2012 ).
[59]
Walid Taha. 1999. Multi-stage programming: Its theory and applications. PhD Thesis. Oregon Graduate Institute of Science and Technology.
[60]
Walid Taha and Michael Florentin Nielsen. 2003. Environment Classiifers. SIGPLAN Not. 38, 1 (Jan. 2003 ), 26-37.
[61]
Walid Taha and Tim Sheard. 1997. Multi-stage programming with explicit annotations. In ACM SIGPLAN Notices, Vol. 32. ACM, 203-217.
[62]
Walid Taha and Tim Sheard. 2000. MetaML and multi-stage programming with explicit annotations. Theor. Comput. Sci. 248, 1-2 ( 2000 ), 211-242.
[63]
Vlad Ureche, Aggelos Biboudis, Yannis Smaragdakis, and Martin Odersky. 2015. Automating Ad Hoc Data Representation Transformations. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2015 ). ACM, New York, NY, USA, 801-820.
[64]
Vlad Ureche, Cristian Talau, and Martin Odersky. 2013. Miniboxing: Improving the Speed to Code Size Tradeof in Parametric Polymorphism Translations. In Proceedings of the ACM SIGPLAN 2013 Conference on Object Oriented Programming, Systems, Languages and Applications (OOPSLA'13).
[65]
David Vandevoorde and Nicolai M. Josuttis. 2002. C++ Templates. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
[66]
Takahisa Watanabe and Yukiyoshi Kameyama. 2017. Program Generation for ML Modules (Short Paper). In Proceedings of the ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (Los Angeles, CA, USA) ( PEPM '18). Association for Computing Machinery, New York, NY, USA, 60-66. htps://doi.org/10.1145/3162072
[67]
Edwin Westbrook, Mathias Ricken, Jun Inoue, Yilong Yao, Tamer Abdelatif, and Walid Taha. 2010. Mint: Java multi-stage programming using weak separability. ACM Sigplan Notices 45, 6 ( 2010 ), 400-411.
[68]
Jeremy Yallop and Oleg Kiselyov. 2019. Generating Mutually Recursive Definitions. In Proceedings of the 2019 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (Cascais, Portugal) (PEPM 2019 ). Association for Computing Machinery, New York, NY, USA, 75-81. htps://doi.org/10.1145/3294032.3294078
[69]
Jeremy Yallop and Leo White. 2015. Modular macros. In OCaml Users and Developers Workshop.

Cited By

View all
  • (2023)Multi-Stage Vertex-Centric Programming for Agent-Based SimulationsProceedings of the 22nd ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3624007.3624057(100-112)Online publication date: 22-Oct-2023
  • (2023)Generalizing Bulk-Synchronous Parallel Processing for Data Science: From Data to Threads and Agent-Based SimulationsProceedings of the ACM on Management of Data10.1145/35892961:2(1-28)Online publication date: 20-Jun-2023
  • (2021)Type-safe generation of modules in applicative and generative stylesProceedings of the 20th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3486609.3487209(184-196)Online publication date: 17-Oct-2021

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GPCE 2020: Proceedings of the 19th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
November 2020
136 pages
ISBN:9781450381741
DOI:10.1145/3425898
  • General Chair:
  • Martin Erwig,
  • Program Chair:
  • Jeff Gray
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 November 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Multi-stage programming
  2. Squid
  3. Staged classes

Qualifiers

  • Research-article

Conference

GPCE '20
Sponsor:
GPCE '20: Concepts and Experiences
November 16 - 17, 2020
Virtual, USA

Acceptance Rates

Overall Acceptance Rate 56 of 180 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)34
  • Downloads (Last 6 weeks)0
Reflects downloads up to 13 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Multi-Stage Vertex-Centric Programming for Agent-Based SimulationsProceedings of the 22nd ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3624007.3624057(100-112)Online publication date: 22-Oct-2023
  • (2023)Generalizing Bulk-Synchronous Parallel Processing for Data Science: From Data to Threads and Agent-Based SimulationsProceedings of the ACM on Management of Data10.1145/35892961:2(1-28)Online publication date: 20-Jun-2023
  • (2021)Type-safe generation of modules in applicative and generative stylesProceedings of the 20th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3486609.3487209(184-196)Online publication date: 17-Oct-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media