skip to main content
research-article

Papiercraft: A gesture-based command system for interactive paper

Published: 19 January 2008 Publication History

Abstract

Paper persists as an integral component of active reading and other knowledge-worker tasks because it provides ease of use unmatched by digital alternatives. Paper documents are light to carry, easy to annotate, rapid to navigate, flexible to manipulate, and robust to use in varied environments. Interactions with paper documents create rich webs of annotation, cross reference, and spatial organization. Unfortunately, the resulting webs are confined to the physical world of paper and, as they accumulate, become increasingly difficult to store, search, and access. XLibris [Schilit et al. 1998] and similar systems address these difficulties by simulating paper with tablet PCs. While this approach is promising, it suffers not only from limitations of current tablet computers (e.g., limited screen space) but also from loss of invaluable paper affordances.
In this article, we describe PapierCraft, a gesture-based command system that allows users to manipulate digital documents using paper printouts as proxies. Using an Anoto [Anoto 2002] digital pen, users can draw command gestures on paper to tag a paragraph, e-mail a selected area, copy selections to a notepad, or create links to related documents. Upon pen synchronization, PapierCraft executes the commands and presents the results in a digital document viewer. Users can then search the tagged information and navigate the web of annotated digital documents resulting from interactions with the paper proxies. PapierCraft also supports real time interactions across mix-media, for example, letting users copy information from paper to a Tablet PC screen. This article presents the design and implementation of the PapierCraft system and describes user feedback from initial use.

References

[1]
Adler, A., Gujar, A., Harrison, L. B., O'Hara, K. and Sellen, A. 1998. A diary study of work-related reading: design implications for digital reading devices. Proceedings of the 1998 ACM Conference on Human Factors in Computer Systems (CHI'98), ACM, New York, pp. 241--248.
[2]
Alvarado, C. and Davis, R. 2004. SketchREAD: A multi-domain sketch recognition engine. Proceedings of UIST'04, ACM, New York, pp. 23--32.
[3]
Anoto. 2002. Development guide for Service Enabled by Anoto Functionality.
[4]
Arai, T., Aust, D., and Hudson, S. E. 1997. PaperLink: A technique for hyperlinking from real paper to electronic content. In Proceedings of the 1997 ACM Conference on Human Factors in Computer System (CHI'97), ACM, New York, pp. 327--334.
[5]
Avrahami, D., Hudson, S. E., Moran, T. P., and Williams, B. D. 2001. Guided gesture support in the paper PDA. In Proceedings of UIST'01. ACM, New York, pp. 197--198.
[6]
Bailey, P. B. and Konstan, A. J. 2003. Are informal tools better?: Comparing DEMAIS, pencil and paper, and authorware for early multimedia design. In Proceedings of the 2003 ACM Conference on Human Factors in Computer Systems (CHI'03), ACM, New York, pp. 313--320.
[7]
Bailey, P. B., Konstan, A. J., and Carlis, V. J. 2001. DEMAIS: Designing multimedia applications with interactive storyboards. In Proceedings of ACM Multimedia'01, ACM, New York, pp. 241--250.
[8]
Carzaniga, A., Rosenblum, D. S., and Wolf, A. L. 2001. Design and evaluation of a wide-area event notification service. ACM Trans. Comput. Syst. 19, 3, 332--383.
[9]
Cohen, P. R. and McGee, D. R. 2004. Tangible multimodal interfaces for safety-critical applications. Commun. ACM 47, 1, 41--46.
[10]
Cunha, P. C. 2004. The I-Book: A freely interpretive notebook for biologists. In Proceedings of Participatory Design Conference'04. pp. 159--162.
[11]
Dymetman, M. and Copperman, M. 1998. Intelligent paper. In Proceedings of EP'98, pp. 392--406.
[12]
Ebrahimi, R. 2005. Reading V2.0 : Next generation reading Systems and the PENHance Plug-in, unpublished honors thesis, University of California, San Diego, CA.
[13]
Google. 2005. Google Books, http://books.google.com.
[14]
Guimbretiere, F. 2003. Paper augmented digital documents. In Proceedings of UIST'03. ACM, New York, pp. 51--60.
[15]
Guimbretière, F., Stone, M., and Winograd, T. 2001. Fluid interaction with high-resolution wall-size displays. In Proceedings of UIST'01. ACM, New York, pp. 21--30.
[16]
Hardock, G., Kurtenbach, G., and Buxton, W. 1993. A marking based interface for collaborative writing. In Proceedings of UIST'93. ACM, New York, pp. 259--266.
[17]
Heiner, J. M., Hudson, S. E., and Tanaka, K. 1999. Linking and messaging from real paper in the paper PDA. In Proceedings of UIST'99. ACM, New York, pp. 179--186.
[18]
Hinckley, K., Baudisch, P., Ramos, G., and Guimbretiere, F. 2005. Design and analysis of delimiters for selection-action pen gesture phrases in scriboli. In Proceedings of the 2005 ACM Conference on Human Factors in Computer Systems (CHI'05). ACM, New York, pp. 451--460.
[19]
Hinckley, K., Ramos, G., Guimbretiere, F., Baudisch, P., and Smith, M. 2004. Stitching: Pen gestures that span multiple displays. In Proceedings of AVI'04. pp. 23--31.
[20]
IBM. 1998. CrossPadd, http://www.research.ibm.com/electricInk/.
[21]
Johnson, W., Jellinek, H., Leigh Klotz, J., Rao, R., and Card, S. K. 1993. Bridging the paper and electronic worlds: The paper user interface. In Proceedings of the 1993 ACM Conference on Human Factors in Computer Science (CHI'93). ACM, New York, pp. 507--512.
[22]
Kara Burak, L. and Stahovich, F. T. 2004. Hierarchical parsing and recognition of hand-sketched diagrams. In Proceedings of UIST'04. ACM, New York, pp. 13--22.
[23]
Khan, F. 1993. A survey of note-taking practices. Tech Rep. HPL-93-107. Personal Systems Laboratory, HP Laboratory.
[24]
Klemmer, S. R., Graham, J., Wolff, G. J., and Landay, J. A. 2003. Books with voices: Paper transcripts as a physical interface to oral histories. In Proceedings of the 2003 ACM Conference on Human Factors in Computer Science (CHI'03). ACM, New York, pp. 89--96.
[25]
Klemmer, S. R., Newman, M. W., Farrell, R., Bilezikjian, M., and Landay, J. A. 2001. The designers' outpost: A tangible interface for collaborative web site. In Proceedings of UIST'01. ACM, New York, pp. 1--10.
[26]
Koike, H., Sato, Y., Kobayashi, Y., Tobita, H., and Kobayashi, M. 2000. Interactive textbook and interactive Venn diagram: Natural and intuitive interfaces on augmented desk system. In Proceedings of the 2000 ACM Conference on Human Factors in Computer Science (CHI'00). ACM, New York, pp. 121--128.
[27]
Landay, A. J. and Myers, A. B. 1995. Interactive sketching for the early stages of user interface design. In Proceedings of the 1995 ACM Conference on Human Factors in Computer Science (CHI'95). ACM, New York, pp. 43--50.
[28]
LeapFrog. 2005. Fly Pen, http://www.leapfrog.com.
[29]
Lee, J. C., Dietz, P. H., Maynes-Aminzade, D., Raskar, R., and Hudson, S. E. 2004. Automatic projector calibration with embedded light sensors. In Proceedings of UIST'04. ACM, New York, pp. 123--126.
[30]
Li, Y., Hinckley, K., Guan, Z., and Landay, J. A. 2005. Experimental analysis of mode switching techniques in pen-based user interfaces. In Proceedings of the 2005 ACM Conference on Human Factors in Computer Science (CHI'05). ACM, New York, pp. 461--470.
[31]
Liao, C., Guimbretière, F., and Hinckley, K. 2005. PapierCraft: A command system for interactive paper. In Proceedings of UIST05. ACM, New York, pp. 241--244.
[32]
Liao, C., Guimbretière, F., and Loeckenhoff, C. E. 2006. Pentop feedback for paper-based interfaces. In Proceedings of UIST'06. ACM, New York, pp. 211--220.
[33]
Lin, J., Newman, W. M., Hong, I. J., and Landay, A. J. 2000. DENIM: Finding a tighter fit between tools and practice for Web site design. In Proceedings of the 2000 ACM Conference on Human Factors in Computer Science (CHI'00). ACM, New York, pp. 510--517.
[34]
Logitech. 2004. IO Digital Pen, http://www.logitech.com.
[35]
Mackay, W. and Pagani, D. 1994. Video mosaic: Laying out time in a physical space. In Proceedings of MM'94. pp. 165--172.
[36]
Mackay, W. E., Fayard, A.-L., Frobert, L., and Médini, L. 1998. Reinventing the familiar: Exploring an augmented reality design space for air traffic control. In Proceedings of the 1998 ACM Conference on Human Factors in Computer Science (CHI'98). ACM, New York, pp. 558--565.
[37]
Mackay, W. E., Pagani, D. S., Faber, L., Inwood, B., Launiainen, P., Brenta, L., and Pouzol, V. 1995. Ariel: Augmenting paper engineering drawings. In Proceedings of the 1995 ACM Conference on Human Factors in Computer Science (CHI'95). ACM, New York, pp. 421--422.
[38]
Mackay, W. E., Pothier, G., Letondal, C., Bøegh, K., and Sørensen, H. E. 2002. The missing link: Augmenting biology laboratory notebooks. In Proceedings of UIST'02. ACM, New York, pp. 41--50.
[39]
Mansouri-Samani, M. and Sloman, M. 1997. GEM: A generalised event monitoring language for distributed systems. IEE/IOP/BCS Distrib. Syst. Eng. J. 4, 25.
[40]
Marshall, C. C., Price, M. N., Golovchinsky, G., and Schilit, B. N. 1999. Introducing a digital library reading appliance into a reading group. In Proceedings of the 4th ACM Conference on Digital Libraries. ACM, New York, pp. 77--84.
[41]
Microsoft. 2003. OneNote, http://www.microsoft.com/onenote.
[42]
Microsoft. 2005. Microsoft Windows XP Tablet PC Edition 2005 Recognizer Pack.
[43]
Miller, R. C. and Myers, B. A. 1999. Synchronizing clipboards of multiple computers. In Proceedings of UIST'99. ACM, New York, pp. 65--66.
[44]
Norrie, M. C. and Signer, B. 2003. Switching over to paper: A new web channel. In Proceedings of the Web Information Systems Engineering'03, pp. 209--218.
[45]
O'Hara, K. and Sellen, A. 1997. A comparison of reading paper and on-line documents. In Proceedings of the ACM 1997 Conference on Human Factors in Computer Science CHI'97. ACM, New York, pp. 335--342.
[46]
Olsen, R. D., Jr., Taufer, T., and Fails, J. A. 2004. ScreenCrayons: Annotating anything. In Proceedings of UIST'04. ACM, New York, pp. 165--174.
[47]
Price, M. N., Golovchinsky, G., and Schilit, B. N. 1998. Linking by inking: Trailblazing in a paper-like hypertext. In Proceedings of the Conference on Hypertext and Hypermedia, pp. 30--39.
[48]
Rekimoto, J. 1997. Pick-and-drop: A direct manipulation technique for multiple computer environments. In Proceedings of UIST'97. ACM, New York, pp. 31--39.
[49]
Rekimoto, J. 1999. Time-machine computing: A time-centric approach for the information environment. In Proceedings of UIST'99. ACM, New York, pp. 45--54.
[50]
Rekimoto, J. and Saitoh, M. 1999. Augmented surfaces: A spatially continuous work space for hybrid computing environments. In Proceedings of the 1999 ACM Conference on Human Factors in Computer Science (CHI'99). ACM, New York, pp. 378--385.
[51]
Saund, E., Fleet, D., Larner, D., and Mahoney, J. 2003. Perceptually-supported image editing of text and graphics. In Proceedings of UIST'03. ACM, New York, pp. 183--192.
[52]
Saund, E. and Lank, E. 2003. Stylus input and editing without prior selection of mode. In Proceedings of UIST'03. ACM, New York, pp. 213--216.
[53]
Schilit, B. N., Golovchinsky, G., and Price, M. N. 1998. Beyond paper: Supporting active reading with free form digital ink annotations. In Proceedings of the ACM 1998 Conference on Human Factors in Computer Science (CHI'98). ACM, New York, pp. 249--256.
[54]
Schön, D. A. 1983. The Reflective Practitioner: How Professionals Think in Action. Temple Smith, London, UK.
[55]
Schön, D. A. and Wiggins, G. 1992. Kinds of seeing and their functions in designing. Des. Stud. 13, 2, 135--156.
[56]
Sellen, A. J. and Harper, R. H. R. 2001. The Myth of the Paperless Office. 1st ed. MIT press, Cambridge, MA.
[57]
Sensiva. 2005. http://www.sensiva.com/.
[58]
Song, H., Guimbretière, F., Hu, C., and Lipson, H. 2006. ModelCraft: Capturing freehand annotations and edits on physical 3D models. In Proceedings of UIST'06. ACM, New York, pp. 13--22.
[59]
Wellner, P. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7, 87--96.
[60]
Wilcox, L. D., Schilit, B. N., and Sawhney, N. 1997. Dynomite: A dynamically organized ink and audio notebook. In Proceedings of the 1997 ACM Conference on Human Factors in Computer Science (CHI'97). ACM, New York, pp. 186--193.
[61]
Yeh, R. B., Liao, C., Klemmer, S. R., Guimbretière, F., Lee, B., Kakaradov, B., Stamberger, J., and Paepcke, A. 2006. ButterflyNet: A mobile capture and access system for field biology research. In Proceedings of the 2006 ACM Conference on Human Factors in Computer Science (CHI'06). ACM, New York, pp. 571--580.
[62]
Zeleznik, R. and Miller, T. 2006. Fluid inking: Augmenting the medium of free-form inking with gestures. In Proceedings of Graphics Interface '06, pp. 155-162.
[63]
Zhao, S. and Balakrishnan, R. 2004. Simple vs. compound mark hierarchical marking menus. In Proceedings of UIST'04. ACM, New York, pp. 33--42.

Cited By

View all
  • (2024)OptiBasePen: Mobile Base+Pen Input on Passive Surfaces by Sensing Relative Base Motion Plus Close-Range Pen PositionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676467(1-9)Online publication date: 13-Oct-2024
  • (2022)The Hybrid Stylus: A Multi-Surface Active Stylus for Interacting with and Handwriting on Paper, Tabletop Display or BothSensors10.3390/s2218705822:18(7058)Online publication date: 18-Sep-2022
  • (2022)Dually Noted: Layout-Aware Annotations with Smartphone Augmented RealityProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502026(1-15)Online publication date: 29-Apr-2022
  • Show More Cited By

Index Terms

  1. Papiercraft: A gesture-based command system for interactive paper

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Computer-Human Interaction
    ACM Transactions on Computer-Human Interaction  Volume 14, Issue 4
    January 2008
    204 pages
    ISSN:1073-0516
    EISSN:1557-7325
    DOI:10.1145/1314683
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 January 2008
    Accepted: 01 June 2007
    Revised: 01 May 2007
    Received: 01 May 2006
    Published in TOCHI Volume 14, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Paper interfaces
    2. distributed systems
    3. gesture-based interfaces
    4. marking interfaces
    5. paper-augmented digital documents
    6. tablet computers
    7. ubiquitous computing

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)59
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)OptiBasePen: Mobile Base+Pen Input on Passive Surfaces by Sensing Relative Base Motion Plus Close-Range Pen PositionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676467(1-9)Online publication date: 13-Oct-2024
    • (2022)The Hybrid Stylus: A Multi-Surface Active Stylus for Interacting with and Handwriting on Paper, Tabletop Display or BothSensors10.3390/s2218705822:18(7058)Online publication date: 18-Sep-2022
    • (2022)Dually Noted: Layout-Aware Annotations with Smartphone Augmented RealityProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502026(1-15)Online publication date: 29-Apr-2022
    • (2021)Contributos para um Sistema de Anotações no Contexto da Comunicação CientíficaProceedings of the 10th International Conference on Digital and Interactive Arts10.1145/3483529.3483703(1-9)Online publication date: 13-Oct-2021
    • (2020)PapercodeAdjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379350.3416191(77-79)Online publication date: 20-Oct-2020
    • (2020)Chameleon: Bringing Interactivity to Static Digital DocumentsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376559(1-13)Online publication date: 21-Apr-2020
    • (2020)texSketch: Active Diagramming through Pen-and-Ink AnnotationsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376155(1-13)Online publication date: 21-Apr-2020
    • (2020)A Method of Safe and Fast Bluetooth Connection and Energy Saving for Educational Environment2020 6th International Symposium on System and Software Reliability (ISSSR)10.1109/ISSSR51244.2020.00022(90-94)Online publication date: Oct-2020
    • (2019)Bridging the Gap Between the Digital and Print Reading ExperienceInternational Journal of Mobile Human Computer Interaction10.4018/IJMHCI.201910010211:4(16-30)Online publication date: Oct-2019
    • (2019)HoloDocProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300917(1-14)Online publication date: 2-May-2019
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media