skip to main content
10.1145/1960275.1960279acmconferencesArticle/Chapter ViewAbstractPublication PagesmodularityConference Proceedingsconference-collections
research-article

MetaLexer: a modular lexical specification language

Published: 21 March 2011 Publication History

Abstract

Compiler toolkits make it possible to rapidly develop compilers and translators for new programming languages. Although there exist elegant toolkits for modular and extensible parsers, compiler developers must often resort to ad-hoc solutions when extending or composing lexers. This paper presents MetaLexer, a new modular lexical specification language and associated tool.
MetaLexer allows programmers to define lexers in a modular fashion. MetaLexer modules can be used to break the lexical specification of a language into a collection smaller modular lexical specifications. Control is passed between the modules using the concept of meta-tokens and meta-lexing. MetaLexer modules are also extensible.
MetaLexer has two key features: it abstracts lexical state transitions out of semantic actions and it makes modules extensible by introducing multiple inheritance.
We have constructed a MetaLexer tool which converts MetaLexer specifications to the popular JFlex lexical specification language and we have used our tool to create lexers for three real programming languages and their extensions: AspectJ (and two AspectJ extensions), MATLAB (and the AspectMatlab extension), and MetaLexer itself. The new specifications are easier to read, are extensible, and require much less action code than the originals.

References

[1]
AspectBench Group. abc extensions. http://www.aspectbench.org/extensions.
[2]
P. Avgustinov, A. S. Christensen, L. Hendren, S. Kuzins, J. Lhoták, O. Lhoták, O. de Moor, D. Sereni, G. Sittampalam, and J. Tibble. abc : An extensible AspectJ compiler. In AOSD 2005: Proceedings of the 4th International Conference on Aspect-Oriented Software Development, pages 87--98, New York, NY, USA, 2005. ACM.
[3]
H. Behrens, M. Clay, S. Efftinge, M. Eysholdt, P. Friese, J. Köhlein, K. Wannheden, S. Zarnekow, and contributors. XText user guide. http://www.eclipse.org/Xtext/documentation/1_0_1/xtext.pdf, 2010.
[4]
C. Brabrand, M. I. Schwartzbach, and M. Vanggaard. The metafront system: Extensible parsing and transformation. In Proceedings of the 3rd ACM SIGPLAN Workshop on Language Descriptions, Tools and Applications, LDTA 2003, volume 82(3), pages 592--611, 2003.
[5]
M. Bravenboer, Éric Tanter, and E. Visser. Declarative, formal, and extensible syntax definition for AspectJ. In OOPSLA 2006: Proceedings of the 21st Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications, pages 209--228, New York, NY, USA, 2006. ACM.
[6]
M. Bravenboer and E. Visser. Concrete syntax for objects. domain-specific language embedding and assimilation without restrictions. In OOPSLA 2004: Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 365--383, New York, NY, USA, 2004. ACM.
[7]
M. Brukman and A. C. Myers. PPG: a parser generator for extensible grammars. http://www.cs.cornell.edu/Projects/polyglot/ppg.html.
[8]
A. Casey. The MetaLexer lexer specification language. Master's thesis, McGill University, Montreal, QC, CA, 2009.
[9]
W. Cazzola and D. Poletti. Dsl evolution through composition. In Proceedings of the 7th ECOOP Workshop on Reflection, AOP and Meta-Data for Software Evolution, page to appear, 2010.
[10]
S. Dmitriev. Language oriented programming: The next programming paradigm. http://www.onboard.jetbrains.com/is1/articles/04/10/lop/.
[11]
T. Ekman and G. Hedin. The Jastadd extensible Java compiler. SIGPLAN Not., 42(10):1--18, 2007.
[12]
T. Ekman and G. Hedin. The JastAdd system: Modular extensible compiler construction. Science of Computer Programming, 69(1-3):14--26, Dec 2007.
[13]
GNU. Flex: The fast lexical analyzer. http://www.gnu.org/software/flex/.
[14]
R. Grimm. Better extensibility through modular syntax. In PLDI 2006: Proceedings of the 2006 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 38--51, New York, NY, USA, 2006. ACM.
[15]
H. Grönniger, H. Krahn, B. Rumpe, M. Schindler, and S. Völkel. MontiCore: A framework for the development of textual domain specific languages. In ICSE Companion 2008: Companion of the 30th International Conference on Software Engineering, pages 925--926, New York, NY, USA, 2008. ACM.
[16]
L. Hendren, O. de Moor, and A. S. Christensen. The abc scanner and parser. Technical report, Programming Tools Group, Oxford University and the Sable research group, McGill University, Sep 2004. http://abc.comlab.ox.ac.uk/documents/scannerandparser.pdf.
[17]
M. Hirzel and R. Grimm. Jeannie: Granting Java Native Interface developers their wishes. ACM SIGPLAN Notices, 42(10):19--38, 2007.
[18]
L. Kats, K. T. Kalleberg, and E. Visser. Generating editors for embedded languages. Technical Report Series TUD-SERG-2008-006, Delft University of Technology, Software Engineering Research Group, 2008. http://swerl.tudelft.nl/twiki/pub/Main/TechnicalReports/TUD-SERG-2008-006.pdf.
[19]
B. W. Kernighan and D. M. Ritchie. The C Programming Language. Prentice-Hall, Englewood Cliffs, New Jersey, 1978.
[20]
G. Kiczales, E. Hilsdale, J. Hugunin, M. Kersten, J. Palm, and W. G. Griswold. An overview of AspectJ. In ECOOP 2001: Object-Oriented Programming, pages 327--353. Springer-Verlag, 2001.
[21]
G. Klein. JFlex: The fast scanner generator for Java. http://jflex.de/.
[22]
H. Krahn, B. Rumpe, and S. Völkel. Efficient editor generation for compositional DSLs in Eclipse. In Proceedings of the 7th OOPSLA Workshop on Domain-Specific Modeling, 2007.
[23]
M. Lesk and E. Schmidt. Lex: A lexical analyzer generator. Comp. Sci. Tech. Rep. 39, Bell Laboratories, Oct 1975.
[24]
MathWorks. Matlab. http://www.mathworks.com/products/matlab/.
[25]
Microsoft. ASP. http://www.asp.net/.
[26]
T. Parr. Composite grammars. http://www.antlr.org/wiki/display/ANTLR3/Composite+Grammars.
[27]
T. Parr. The Definitive ANTLR Reference: Building Domain-Specific Languages. Pragmatic Bookshelf, Raleigh, NC, 2007.
[28]
Sable Lab. Mclab. http://www.sable.mcgill.ca/mclab/.
[29]
A. C. Schwerdfeger and E. R. V. Wyk. Verifiable composition of deterministic grammars. In PLDI 2009: Proceedings of the 2009 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 199--210, New York, NY, USA, 2009. ACM.
[30]
Sun Microsystems. JSP. http://java.sun.com/products/jsp/.
[31]
Technische Universität München. CUP: LALR parser generator in Java. http://www2.cs.tum.edu/projects/cup/.
[32]
W3C. Cascading Style Sheets. http://www.w3.org/Style/CSS/.
[33]
W3C. Scripts in HTML4. http://www.w3.org/TR/html4/interact/scripts.html.
[34]
E. R. V. Wyk and A. C. Schwerdfeger. Context-aware scanning for parsing extensible languages. In GPCE 2007: Proceedings of the 6th International Conference on Generative Programming and Component Engineering, pages 63--72, New York, NY, USA, 2007. ACM.

Cited By

View all
  • (2013)Object GrammarsSoftware Language Engineering10.1007/978-3-642-36089-3_2(4-23)Online publication date: 2013
  • (2012)McSAFProceedings of the 26th European conference on Object-Oriented Programming10.1007/978-3-642-31057-7_7(132-155)Online publication date: 11-Jun-2012
  • (2011)Staged static techniques to efficiently implement array copy semantics in a MATLAB JIT compilerProceedings of the 20th international conference on Compiler construction: part of the joint European conferences on theory and practice of software10.5555/1987237.1987242(22-41)Online publication date: 26-Mar-2011
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
AOSD '11: Proceedings of the tenth international conference on Aspect-oriented software development
March 2011
326 pages
ISBN:9781450306058
DOI:10.1145/1960275
  • General Chair:
  • Paulo Borba,
  • Program Chair:
  • Shigeru Chiba
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

  • Centro de Informatica - UFPE: Centro de Informatica - UFPE
  • CAPES: Coordençãao de Aperfeiçoamento de Pessoal de Nível Superior
  • CNPq: Conselho Nacional de Desenvolvimento Cientifico e Tecn
  • FINEP: Financiadora de Estudos e Projetos
  • PUC-Rio
  • FACEPE: Fundacao de Amparo a Ciencia e Tecnologia do Estado de Pernambuco

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 March 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. lexer
  2. meta-lexing
  3. modular lexing
  4. modular scanning
  5. scanner

Qualifiers

  • Research-article

Conference

AOSD'11

Acceptance Rates

Overall Acceptance Rate 41 of 139 submissions, 29%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2013)Object GrammarsSoftware Language Engineering10.1007/978-3-642-36089-3_2(4-23)Online publication date: 2013
  • (2012)McSAFProceedings of the 26th European conference on Object-Oriented Programming10.1007/978-3-642-31057-7_7(132-155)Online publication date: 11-Jun-2012
  • (2011)Staged static techniques to efficiently implement array copy semantics in a MATLAB JIT compilerProceedings of the 20th international conference on Compiler construction: part of the joint European conferences on theory and practice of software10.5555/1987237.1987242(22-41)Online publication date: 26-Mar-2011
  • (2011)McLABProceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion10.1145/2048147.2048203(195-196)Online publication date: 22-Oct-2011
  • (2011)Staged Static Techniques to Efficiently Implement Array Copy Semantics in a MATLAB JIT CompilerCompiler Construction10.1007/978-3-642-19861-8_3(22-41)Online publication date: 2011

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media