Skip to main content
SpringerLink
Log in

Defining the Syntax and Semantics of Natural Visual Languages

  • Conference paper
  • First Online:
Applications of Graph Transformations with Industrial Relevance (AGTIVE 1999)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 1779))

  • 572 Accesses

Abstract

To bridge the gap between paper and electronic forms of documents, computers must be able to recognize and generate diagrams as well as text. Diagrams used in society are expressed in a variety of notations, which we call natural visual languages. Examples include notations used for mathematics, music, engineering drawings and architecture. These visual languages do not have fixed, formal definitions, but evolve through use in society. This paper examines the use of graph transformation in processing natural visual languages, describing the difficult problems in this domain, existing graph transformation work in this area, and competing methods. Many problems have not been adequately addressed by any technique. The use of graph transformation is appropriate, since the representation and manipulation of spatial and logical relationships is central to the computation.

This research is supported by Canada’s Natural Sciences and Engineering Research Council.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. F. Arefi, C. Hughes, D. Workman, “Automatically Generating Visual Syntax-Directed Editors,” Communications of the ACM, 33(3), March 1990, pp. 349–360.

    Article  Google Scholar 

  2. S. Baumann, “A Simplified Attributed Graph Grammar for High-Level Music Recognition,” Proc. Third Intl. Conf. on Document Analysis and Recognition, Montreal, Canada, Aug. 1995, pp. 1080–1083.

    Google Scholar 

  3. D. Blostein, “General Diagram-Recognition Methodologies,” in Graphics Recognition — Methods and Applications, Eds. R. Kasturi, K. Tombre, LNCS Vol. 1072, Springer Verlag, 1996, pp. 106–122.

    Google Scholar 

  4. D. Blostein, “Application of Graph Rewriting to Document Image Analysis,” Proc. Theory and Application of Graph Transformations — TAGT’98, Paderborn, Germany, Nov. 1998, pp. 16–23.

    Google Scholar 

  5. D. Blostein, A. Schürr, “Computing with Graphs and Graph Transformation,” Software Practice and Experience, 29(3), 1999, pp. 197–217.

    Article  Google Scholar 

  6. H. Bunke, “Attributed Programmed Graph Grammars and Their Application to Schematic Diagram Interpretation,” IEEE Trans. Pattern Analysis and Machine Intelligence 4(6), Nov. 1982, pp. 574–582.

    Article  MATH  Google Scholar 

  7. P. Chou, “Recognition of Equations Using a Two-Dimensional Stochastic Context-Free Grammar,” Proc. SPIE Visual Communications and Image Processing IV, Philadelphia PA, Nov. 1989, pp.852–863.

    Google Scholar 

  8. J. Cordy, C. Halpern-Hamu, E. Promislow, “TXL: A Rapid Prototyping System for Programming Language Dialects”, Computer Languages, 16(1), Jan. 1991, pp. 97–107.

    Article  Google Scholar 

  9. D. Dori, A. Pnueli, “The Grammar of Dimensions in Machine Drawings,” Computer Vision, Graphics and Image Processing, 42, pp. 1–18, 1988.

    Article  Google Scholar 

  10. H. Fahmy, D. Blostein, “A Graph Grammar Programming Style for Recognition of Music Notation,” Machine Vision and Applications, 6(2), 1993, pp. 83–99.

    Article  Google Scholar 

  11. H. Fahmy, D. Blostein, “A Graph-Rewriting Paradigm for Discrete Relaxation: Application to Sheet-Music Recognition,” Intl. Journal of Pattern Recognition and Artificial Intelligence, 12(6), Sept. 1998, pp. 763–799.

    Article  Google Scholar 

  12. H. Göttler, “Diagram Editors = Graphs + Attributes + Graph Grammars,” Intl. Journal of Man-Machine Studies, 37(4), Oct. 1992, pp. 481–502.

    Article  Google Scholar 

  13. A. Grbavec, D. Blostein, “Mathematics Recognition Using Graph Rewriting,” Third Intl. Conf. Document Analysis and Recognition, Montreal, Canada, Aug. 1995, pp. 417–421.

    Google Scholar 

  14. L. Haken, D. Blostein, “A New Algorithm for Horizontal Spacing of Printed Music,” Intl. Computer Music Conf, Banff, Sept. 1995, pp. 118–119.

    Google Scholar 

  15. International Journal on Document Analysis and Recognition, Springer Verlag.

    Google Scholar 

  16. D. Isenor, S. Zaky, “Fingerprint Identification Using Graph Matching,” Pattern Recognition, 19(2), 1986, pp. 113–122.

    Article  Google Scholar 

  17. D. Knuth, “Mathematical Typography,” Bulletin of the American Mathematical Society, 1(2), March 1979.

    Google Scholar 

  18. G. Kopec, P. Chou, “Document Image Decoding Using Markov Source Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, 16(6), June 1994, pp. 602–617.

    Article  Google Scholar 

  19. S. Lavirotte, L. Pottier, “Optical Formula Recognition,” Fourth Intl. Conf. on Document Analysis and Recognition, Ulm, Germany, Aug. 1997, pp. 357–361.

    Google Scholar 

  20. G. Lohse, K. Biolsi, N. Walker, H. Ruetter, “A Classification of Visual Representations,” Communications of the ACM, 37(4), Dec. 1994, pp. 36–49.

    Article  Google Scholar 

  21. K. Marriott, B. Meyer, K. Wittenburg, “A Survey of Visual Language Specification and Recognition,” in Visual Language Theory, Eds. K. Marriott, B. Meyer, Springer Verlag 1998, pp. 5–85.

    Google Scholar 

  22. B. Messmer, H. Bunke, “Clustering and Error-Correcting Matching of Graphs for Learning and Recognition of Symbols in Engineering Drawings,” DAS 96 IAPR Workshop on Document Analysis Systems, Malvern PA, Oct. 1996, pp. 282–296.

    Google Scholar 

  23. M. Minas, “Automatically Generating Environments for Dynamic Diagram Languages,” Proc. 1998 IEEE Symposium on Visual Languages, Halifax, Canada, Sept. 1998, pp. 70–71.

    Google Scholar 

  24. L. O’Gorman, R. Kasturi, Document Image Analysis, IEEE Computer Society Press, 1995.

    Google Scholar 

  25. F. Parmentier, A. Belaïd, “Logical Structure Recognition of Scientific Bibliographic References,” Fourth Intl. Conf. on Document Analysis and Recognition, Ulm, Germany, Aug. 1997, pp. 1072–1076.

    Google Scholar 

  26. B. Pasternak, “Processing Imprecise and Structural Distorted Line Drawings by an Adaptable Drawing Interpretation Kernel,” Proc. IAPR Workshop on Document Analysis Systems, Kaiserslautern, Germany, Oct. 1994, pp. 349–363.

    Google Scholar 

  27. M. Petre, “Why Looking Isn’t Always Seeing: Readership Skills and Graphical Programming,” Communications of the ACM, 38(6), June 1995, pp. 33–44.

    Article  Google Scholar 

  28. A. Pies, Repräsentation und Verarbeitung von Musikalischem Wissen — Eine Attributierte Programmierte Graph-Grammatik zur Erkennung Gedruckter Partituren, Diplomarbeit, DFKI Kaiserslautern, Fachbereich Informatik, Aug. 1994.

    Google Scholar 

  29. B. Poirier, M. Dagenais, “An Interactive System to Extract Structured Text from a Geometrical Representation,” Fourth Intl. Conf. on Document Analysis and Recognition, Ulm, Germany, Aug. 1997, pp. 342–346.

    Google Scholar 

  30. L. Protsko, P. Sorenson, J. Tremblay, D. Schaefer, “Towards the Automatic Generation of Software Diagrams,” IEEE Trans. Software Engineering, 17(1), Jan. 1991, pp. 10–21.

    Article  Google Scholar 

  31. Proc. Annual Symposia on Document Analysis and Information Retrieval, Las Vegas, 1992, 1993, 1994, 1995, 1996, 1997, sponsored by the University of Nevada.

    Google Scholar 

  32. Proc. IAPR Workshops on Document Analysis Systems, Kaiserslautern, Germany, Oct. 1994; Malvern PA, Oct. 1996; Nagano, Japan, Nov. 1998.

    Google Scholar 

  33. Proc. IAPR Workshops on Graphics Recognition, Pennsylvania, USA 1995; Nancy, France, 1997; Jaipur, India, 1999. http://graphics.basit.com/iapr-tcl0/GREC99/

  34. Proc. Intl. Confs. on Document Analysis and Recognition, France, 1991; Japan 1993; Canada 1995; Germany 1997; Japan 1999; sponsored by IAPR and IEEE.

    Google Scholar 

  35. M. A. Rahgozar, R. Cooperman, “A Graph-based Table Recognition System,” Document Recognition III, SPIE Proceedings Series, Vol. 2660, San Jose, California, Jan. 1996, pp. 192–203.

    Google Scholar 

  36. G. Read, Music Notation: A Manual of Modern Practice (Second Edition), Taplinger Publishing, New York, NY, 1979.

    Google Scholar 

  37. J. Serrano, “The Use of Semantic Constraints on Diagram Editors,” Proc. 11th IEEE Symposium on Visual Languages, Darmstadt, Germany, Sept. 1995, pp. 211–216.

    Google Scholar 

  38. A. Smithies, K. Novins, J. Arvo, “A Handwriting-Based Equation Editor,” Proc Graphics Interface’ 99, Kingston, Canada, June, 1999, pp. 84–91.

    Google Scholar 

  39. R. Tamassia, G. Battista, C. Batini, “Automatic Graph Drawing and Readability of Diagrams,” IEEE Trans. Systems, Man and Cybernetics, 18(1), Jan. 1988, pp. 61–79.

    Article  Google Scholar 

  40. R. Wilson, E. Hancock, “Structural Matching by Discrete Relaxation,” IEEE Trans. Pattern Analysis and Machine Intelligence, 19(6), June 1997, pp. 634–648.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dept. Computing and Information Science, Queen’s University, Kingston, Ontario, Canada, K7L 3N6

    Dorothea Blostein

Authors
  1. Dorothea Blostein
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science III, Aachen University of Technology (RWTH), Ahornstr. 55, 52074, Aachen, Germany

    Manfred Nagl  & Manfred Münch  & 

  2. Institute for Software Technology, Department of Computer Science, University of the German Federal Armed Forces Munich, Werner-Heisenberg-Weg 39, 85579, Neubiberg, Germany

    Andreas Schürr

Rights and permissions

Reprints and permissions

Copyright information

© 2000 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Blostein, D. (2000). Defining the Syntax and Semantics of Natural Visual Languages. In: Nagl, M., Schürr, A., Münch, M. (eds) Applications of Graph Transformations with Industrial Relevance. AGTIVE 1999. Lecture Notes in Computer Science, vol 1779. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45104-8_16

Download citation

  • DOI: https://doi.org/10.1007/3-540-45104-8_16

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-67658-4

  • Online ISBN: 978-3-540-45104-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation