skip to main content
10.1145/1027527.1027713acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
Article

Speech, ink, and slides: the interaction of content channels

Published: 10 October 2004 Publication History

Abstract

In this paper, we report on an empirical exploration of digital ink and speech usage in lecture presentation. We studied the video archives of five Master's level Computer Science courses to understand how instructors use ink and speech together while lecturing, and to evaluate techniques for analyzing digital ink. Our interest in understanding how ink and speech are used together is to inform the development of future tools for supporting classroom presentation, distance education, and viewing of archived lectures. We want to make it easier to interact with electronic materials and to extract information from them. We want to provide an empirical basis for addressing challenging problems such as automatically generating full text transcripts of lectures, matching speaker audio with slide content, and recognizing the meaning of the instructor's ink. Our results include an evaluation of handwritten word recognition in the lecture domain, an approach for associating attentional marks with content, an analysis of linkage between speech and ink, and an application of recognition techniques to infer speaker actions.

References

[1]
Abowd, G., Classroom 2000: An experiment with the instrumentation of a living environment. IBM Systems Journal, Voume 38, Number 4, 1999.
[2]
Adler, A, and Davis, R., Speech and Sketching for Multimodal Design, Intelligent User Interfaces'04, pp. 214--216, 2004.
[3]
Altman, E., Chen, Y., and Low, W., Semantic Exploration of Lecture Videos, ACM Multimedia'02 pp.416--417, 2002.
[4]
Anderson, R. J., Anderson, R. E., Hoyer, C. L., and Wolfman, S., A Study of Digital Ink in Lecture Presentation. CHI'04, pp. 567--574, April, 2004.
[5]
Anderson, R. J., Anderson, R. E, Simon, B., Wolfman, S., A., VanDeGrift, T., and Yasuhara, K., "Experiences with a Tablet PC Based Lecture Presentation System in Computer Science Courses," SIGCSE 2004, pp. 56--60, 2004.
[6]
Bacher, C., and Muller, R., Generalized Replay of Multi-Streamed Authored Documents, Proceedings of ED-Media, Freiburg, 1998.
[7]
Bargeron, D., and Moscovich, T., Reflowing digital ink annotation, CHI'03, pp.385--392, 2003.
[8]
Berque, D., Bonewrite, T., and Whitesell, M., Using Pen-Based Computers Across the Computer Science Curriculum, 35th ACM SIGCSE, pp. 61--65, 2004.
[9]
Chu, W-T., and Chen, H-Y., Cross-Media Correlation: A Case Study of Navigated Hypermedia Documents, Multimedia'02, pp. 57--66, 2002.
[10]
Fridland, G., Knipping, L., Rojas, R., E-Chalk: Technical Description, Technical Report B-02-11, FU Berlin, Institut fur Informatik, May 2002.
[11]
Gale, W. A., Church, K. W., and Yarowsky, D. "Using bilingual materials to develop word sense disambiguation methods." Int'l. Conf. on Theoretical and Methodological Issues in Machine Translation, pp.101--112, 1992.
[12]
Gross, M. D., and Do, E. Y., "Drawing on the Back of an Envelope: a framework for interacting with application programs by freehand drawing," Computers & Graphics, 24 pp. 835--849, 2000.
[13]
Jarrett, R., and Su, P., Building Tablet PC Applications, Microsoft Press, 2002.
[14]
Liao, C., Liu, Q., Kimber, D., Chiu, P. Foot, J., and Wilcox, L., Shared Interactive Video Teleconferencing, ACM Multimedia'03, pp. 546--554, 2003.
[15]
Landay, J. A., and Myers, B. A., Sketching Interfaces: Toward More Human Interface Design, IEEE Computer, Vol 34, No. 3, pp 56--64, March 2001
[16]
Lopresti, D., Ink as Multimedia Data, Proceedings of the Fourth Intl. Conference on Information, Systems, Analysis and Synthesis, July 1998, Orlando, FL, pp. 122--128.
[17]
Mukhopadhyay, S., and Smith, B., Passive Capture and Structuring of Lectures, ACM Multimedia '99, Orlando, Fl, pp. 477--487, 1999.
[18]
Plamondon, R., and Srihari, S., N., On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey, IEEE PAMI,. 22(1), pp. 63--84, January 2000.
[19]
Shilman, M., Wei, Z., Raghupathy, S., Simard, P., and Jones, D., Discerning Structure from Freeform Handwritten Notes, ICDAR 2003.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on Multimedia
October 2004
1028 pages
ISBN:1581138938
DOI:10.1145/1027527
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2004

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. digital ink
  2. ink recognition
  3. presentation
  4. speech recognition

Qualifiers

  • Article

Conference

MM04

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)1
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2017)Multimodal speech and pen interfacesThe Handbook of Multimodal-Multisensor Interfaces10.1145/3015783.3015795(403-447)Online publication date: 24-Apr-2017
  • (2017)Combining Speech and Handwriting Modalities for Mathematical Expression RecognitionIEEE Transactions on Human-Machine Systems10.1109/THMS.2017.264785047:2(259-272)Online publication date: Apr-2017
  • (2016)Freeform digital ink annotations in electronic documentsComputers and Graphics10.1016/j.cag.2015.10.01455:C(1-20)Online publication date: 1-Apr-2016
  • (2016)Analysis of Student Perspectives on Using Tablet PCs in Junior and Senior Level Chemical Engineering CoursesRevolutionizing Education with Digital Ink10.1007/978-3-319-31193-7_21(307-319)Online publication date: 19-May-2016
  • (2012)Improving document retrieval using special characteristics of lecture recording documentsProceedings of the 3rd Symposium on Information and Communication Technology10.1145/2350716.2350754(250-259)Online publication date: 23-Aug-2012
  • (2012)Observational study on teaching artifacts created using tablet PCCHI '12 Extended Abstracts on Human Factors in Computing Systems10.1145/2212776.2212809(301-316)Online publication date: 5-May-2012
  • (2011)German Speech RecognitionProceedings of the 2011 10th IEEE/ACIS International Conference on Computer and Information Science10.1109/ICIS.2011.38(201-206)Online publication date: 16-May-2011
  • (2011)A multimodal alignment framework for spoken documentsMultimedia Tools and Applications10.1007/s11042-011-0842-x61:2(353-388)Online publication date: 13-Jul-2011
  • (2009)Integrating corrections into digital ink playbackProceedings of the 17th ACM international conference on Multimedia10.1145/1631272.1631413(781-784)Online publication date: 23-Oct-2009
  • (2008)MultiPresenterProceedings of the 16th ACM international conference on Multimedia10.1145/1459359.1459428(519-528)Online publication date: 26-Oct-2008
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media