skip to main content
10.1145/1281500.1281528acmconferencesArticle/Chapter ViewAbstractPublication PagessiggraphConference Proceedingsconference-collections
Article

Hierarchical parsing and recognition of hand-sketched diagrams

Published: 05 August 2007 Publication History

Abstract

A long standing challenge in pen-based computer interaction is the ability to make sense of informal sketches. A main difficulty lies in reliably extracting and recognizing the intended set of visual objects from a continuous stream of pen strokes. Existing pen-based systems either avoid these issues altogether, thus resulting in the equivalent of a drawing program, or rely on algorithms that place unnatural constraints on the way the user draws. As one step toward alleviating these difficulties, we present an integrated sketch parsing and recognition approach designed to enable natural, fluid, sketch-based computer interaction. The techniques presented in this paper are oriented toward the domain of network diagrams. In the first step of our approach, the stream of pen strokes is examined to identify the arrows in the sketch. The identified arrows then anchor a spatial analysis which groups the uninterpreted strokes into distinct clusters, each representing a single object. Finally, a trainable shape recognizer, which is informed by the spatial analysis, is used to find the best interpretations of the clusters. Based on these concepts, we have built SimuSketch, a sketch-based interface for Matlab's Simulink software package. An evaluation of SimuSketch has indicated that even novice users can effectively utilize our system to solve real engineering problems without having to know much about the underlying recognition techniques.

References

[1]
Fevzi Alimoglu and Ethem Alpaydin. Combining multiple representations for pen-based handwritten digit recognition. ELEKTRIK: Turkish Journal of Electrical Engineering and Computer Sciences, 9(1): 1--12, 2001.
[2]
Christine Alvarado. A Natural Sketching Environment: Bringing the Computer into Early Stages of Mechanical Design. Master thesis, MIT, 2000.
[3]
Christine Alvarado. Dynamically constructed bayesian networks for sketch understanding. Technical report, MIT Project Oxygen Student Workshop Abstracts, 2003.
[4]
Christine Alvarado and Randall Davis. Resolving ambiguities to create a natural sketch based interface. In IJCAI-2001, 2001.
[5]
Ajay Apte, Van Vo, and Takayuki Dan Kimura. Recognizing multistroke geometric shapes: An experimental evaluation. In UIST93, pages 121--128, 1993.
[6]
Chris Calhoun, Thomas F Stahovich, Tolga Kurtoglu, and Levent Burak Kara. Recognizing multi-stroke symbols. In AAAI Spring Symposium on Sketch Understanding, pages 15--23, 2002.
[7]
Gennaro Costagliola and Vincenzo Deufemia. Visual language editors based on lr parsing techniques. In 8th International Workshop on Parsing Technologies (IWPT'03), Nancy, France, 2003.
[8]
Marie-Pierre Dubuisson and Anil K Jain. A modified hausdorff distance for object matching. In 12th International Conference on Pattern Recognition, pages 566--568, Jerusalem, Israel, 1994.
[9]
Lee D Erman, Frederick Hayes-Roth, Victor R Lesser, and D Raj Reddy. The hearsay-ii speech understanding system: Integrating knowldge to resolve uncertainty. Computing Surveys, 12(2):213--253, 1980.
[10]
Michael Fligner, Joseph Verducci, Jeff Bjoraker, and Paul Blower. A new association coefficient for molecular dissimilarity. In The Second Joint Sheffield Conference on Chemoinformatics, Sheffield, England, 2001.
[11]
Manueal J Fonseca, Cesar Pimentel, and Jaoquim A Jorge. Cali-an online scribble recognizer for calligraphic interfaces. In AAAI Spring Symposium on Sketch Understanding, pages 51--58, 2002.
[12]
Manuel J Fonseca and Joaquim A Jorge. Using fuzzy logic to recognize geometric shapes interactively. In Proceedings of the 9th Int. Conference on Fuzzy Systems (FUZZ-IEEE 2000). San Antonio, USA, 2000.
[13]
W Eric L Grimson. The combinatorics of heuristic search termination for object recognition in cluttered environments. IEEE PAMI, 13(9):920--935, 1991.
[14]
Jason I Hong and James A Landay. Satin: A toolkit for informal ink-based applications. In ACM UIST 2000 User Interfaces and Software Technology, pages 63--72, San Diego, CA, 2000.
[15]
Heloise Hse and A. Richard Newton. Sketched symbol recognition using zernike moments. Technical report, EECS, University of California, 2003.
[16]
David W Jacobs. The use of grouping in visual object recognition. Technical Report Technical Report 1023, MIT AI Lab, 1988.
[17]
T D Kimura, A Apte, and S Sengupta. A graphic diagram editor for pen computers. Software Concepts and Tools, pages 82--95, 1994.
[18]
Tolga Kurtoglu and Thomas F Stahovich. Interpreting schematic sketches using physical reasoning. In AAAI Spring Symposium on Sketch Understanding, pages 78--85, 2002.
[19]
Ernst Kussul and Tatyana Baidyk. Improved method of handwritten digit recognition tested on mnist database. In 15th International Conference on Vision Interface, Calgary, Canada, 2002.
[20]
James A Landay and Brad A Myers. Sketching interfaces: Toward more human interface design. IEEE Computer, 34(3):56--64, 2001.
[21]
Y LeCun, L D Jackel, L Bottou, A Brunot, C Cortes, J S Denker, H Drucker, I Guyon, U A Muller, E Sackinger, P Simard, and V Vapnik. Comparison of learning algorithms for handwritten digit recognition. In International Conference on Artificial Neural Networks, pages 53--60, Paris, 1995.
[22]
James Lin, Mark W. Newman, Jason I. Hong, and James A. Landay. Denim: Finding a tighter fit between tools and practice for web site design. In CHI Letters: Human Factors in Computing Systems, pages 510--517. ACM Press, 2000.
[23]
Jennifer Mankoff, Gregory D. Abowd, and Scott E Hudson. Oops: a toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces.Computers and Graphics, 24(6):819--834, 2000.
[24]
Nicholas E Matsakis. Recognition of Handwritten Mathematical Expressions. Master thesis, MIT, 1999.
[25]
Shankar Narayanaswamy. Pen and Speech Recognition in the User Interface for Mobile Multimedia Terminals. Ph.d. thesis, University of California at Berkeley, 1996.
[26]
Omer Faruk Ozer, Oguz Ozun, C Oncel Tuzel, Volkan Atalay, and A Enis Cetin. Vision-based single-stroke character recognition for wearable computing. IEEE Intelligent Systems and Applications, 16(3):33--37, 2001.
[27]
Dean Rubine. Specifying gestures by example. Computer Graphics, 25:329--337, 1991.
[28]
W J Rucklidge. Efficient Visual Recognition Using the Hausdorff Distance. Number 1173 Lecture Notes in computer Science,. Springer-Verlag, Berlin, 1996.
[29]
Eric Saund, James Mahoney, David Fleet, Dan Larner, and Edward Lank. Perceptual organisation as a foundation for intelligent sketch editing. In AAAI Spring Symposium on Sketch Understanding, pages 118--125, 2002.
[30]
Tevfik Metin Sezgin. Generic and HMM based approaches to freehand sketch recognition. Technical report, MIT Project Oxygen Student Workshop Abstracts, 2003.
[31]
Michael Shilman, Hanna Pasula, Stuart Russell, and Richard Newton. Statistical visual language models for ink parsing. In AAAI Spring Symposium on Sketch Understanding, pages 126--132, 2002.
[32]
Jack D Tubbs. A note on binary template matching. Pattern Recognition, 22(4):359--365, 1989.
[33]
H Yasuda, K Takahashi, and T Matsumoto. A discrete HMM for online handwriting recognition. International Journal of Pattern Recognition and Articial Intelligence, 14(5):675--688, 2000.

Cited By

View all
  • (2022)A non-interactive hybrid method for structured offline documents.: Application on the mathematical expression2022 8th International Conference on Information Technology Trends (ITT)10.1109/ITT56123.2022.9863934(145-149)Online publication date: 25-May-2022
  • (2018)The role of grouping in sketched diagram recognitionProceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering10.1145/3229147.3229160(1-12)Online publication date: 17-Aug-2018
  • (2017)DCSR: A Digital Circuit Sketch Recognition System for EducationFrontiers in Pen and Touch10.1007/978-3-319-64239-0_10(137-146)Online publication date: 2-Dec-2017
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGGRAPH '07: ACM SIGGRAPH 2007 courses
August 2007
6166 pages
ISBN:9781450318235
DOI:10.1145/1281500
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 August 2007

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. SimuSketch
  2. pen computing
  3. simulink
  4. sketch understanding
  5. symbol recognition
  6. visual parsing

Qualifiers

  • Article

Conference

SIGGRAPH07
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)1
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)A non-interactive hybrid method for structured offline documents.: Application on the mathematical expression2022 8th International Conference on Information Technology Trends (ITT)10.1109/ITT56123.2022.9863934(145-149)Online publication date: 25-May-2022
  • (2018)The role of grouping in sketched diagram recognitionProceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering10.1145/3229147.3229160(1-12)Online publication date: 17-Aug-2018
  • (2017)DCSR: A Digital Circuit Sketch Recognition System for EducationFrontiers in Pen and Touch10.1007/978-3-319-64239-0_10(137-146)Online publication date: 2-Dec-2017
  • (2016)Offline text and non-text segmentation for hand-drawn diagramsProceedings of the 14th Pacific Rim International Conference on Trends in Artificial Intelligence10.1007/978-3-319-42911-3_32(380-392)Online publication date: 22-Aug-2016
  • (2015)A flexible framework for online document segmentation by pairwise stroke distance learningPattern Recognition10.1016/j.patcog.2014.10.02248:4(1197-1210)Online publication date: 1-Apr-2015
  • (2014)Recognition System for On-Line Sketched Diagrams2014 14th International Conference on Frontiers in Handwriting Recognition10.1109/ICFHR.2014.100(563-568)Online publication date: Sep-2014
  • (2014)MahlerProceedings of the 27th International Conference on Architecture of Computing Systems — ARCS 2014 - Volume 835010.1007/978-3-319-04891-8_8(85-97)Online publication date: 25-Feb-2014
  • (2009)VisionSynapticsProceedings of the 2nd International Conference on Interaction Sciences: Information Technology, Culture and Human10.1145/1655925.1655940(82-85)Online publication date: 24-Nov-2009
  • (2009)VizDrawProceedings of the 6th International Conference on Image Analysis and Recognition10.1007/978-3-642-02611-9_38(377-386)Online publication date: 7-Jul-2009

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media