skip to main content
10.1145/3126594.3126650acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

Markit and Talkit: A Low-Barrier Toolkit to Augment 3D Printed Models with Audio Annotations

Published: 20 October 2017 Publication History

Abstract

As three-dimensional printers become more available, 3D printed models can serve as important learning materials, especially for blind people who perceive the models tactilely. Such models can be much more powerful when augmented with audio annotations that describe the model and their elements. We present Markit and Talkit, a low-barrier toolkit for creating and interacting with 3D models with audio annotations. Makers (e.g., hobbyists, teachers, and friends of blind people) can use Markit to mark model elements and associate then with text annotations. A blind user can then print the augmented model, launch the Talkit application, and access the annotations by touching the model and following Talkit's verbal cues. Talkit uses an RGB camera and a microphone to sense users' inputs so it can run on a variety of devices. We evaluated Markit with eight sighted "makers" and Talkit with eight blind people. On average, non-experts added two annotations to a model in 275 seconds (SD=70) with Markit. Meanwhile, with Talkit, blind people found a specified annotation on a model in an average of 7 seconds (SD=8).

Supplementary Material

suppl.mov (uistf4174.m4v)
Supplemental video

References

[1]
Eric Akaoka, Tim Ginn, and Roel Vertegaal. 2010. DisplayObjects: Prototyping Functional Physical Interfaces on 3D Styrofoam, Paper or Cardboard Models. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (TEI '10), 49--56.
[2]
American Printing House. Product: DNA Twist. Retrieved April 2, 2017 from https://shop.aph.org/webapp/wcs/stores/servlet/Product_DNA Twist_1-08978-00P_10001_11051
[3]
American Printing House. Product: Globe: Tactile and Visual. Retrieved April 2, 2017 from https://shop.aph.org/webapp/wcs/stores/servlet/Product_Globe: Tactile and Visual_1-01551-00P_10001_11051
[4]
Catherine M. Baker, Lauren R. Milne, Jeffrey Scofield, Cynthia L. Bennett, and Richard E. Ladner. 2014. Tactile graphics with a voice. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility (ASSETS '14), 75--82.
[5]
Connelly Barnes, David E. Jacobs, Jason Sanders, Dan B Goldman, Szymon Rusinkiewicz, Adam Finkelstein, and Maneesh Agrawala. 2008. Video Puppetry: A Performative Interface for Cutout Animation. ACM Transactions on Graphics 27, 5: 1--9.
[6]
bld. Textured Earth - Thingiverse. Retrieved April 4, 2017 from http://www.thingiverse.com/thing:17336
[7]
Blender. Free and Open 3D Creation Software. Retrieved April 4, 2017 from https://www.blender.org/
[8]
Quentin Bonnard, Severin Lemaignan, Guillaume Zufferey, Andrea Mazzei, Sebastien Cuendet, Nan Li, Ayberk Ozgur, and Pierre Dillenbourg. 2013. Chilitags 2: Robust Fiducial Markers for Augmented Reality and Robotics.
[9]
Gary R. Bradski and Adrian. Kaehler. 2008. Learning OpenCV?: computer vision with the OpenCV library. O'Reilly.
[10]
John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194: 4--7.
[11]
Craig Brown and Amy Hurst. 2012. VizTouch: automatically generated tactile visualizations of coordinate spaces. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12), 131--138.
[12]
Emeline Brule, Gilles Bailly, Anke Brock, Frederic Valentin, Grégoire Denis, and Christophe Jouffrais. 2016. MapSense: Multi-Sensory Interactive Maps for Children Living with Visual Impairments. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 445--457.
[13]
Erin Buehler, Stacy Branham, Abdullah Ali, Jeremy J Chang, Megan Kelly Hofmann, Amy Hurst, and Shaun K Kane. 2015. Sharing is Caring: Assistive Technology Designs on Thingiverse. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 525--534.
[14]
Erin Buehler, Niara Comrie, Megan Hofmann, Samantha McDonald, and Amy Hurst. 2016. Investigating the Implications of 3D Printing in Special Education. ACM Transactions on Accessible Computing 8, 3: 1--28.
[15]
Erin Buehler, Amy Hurst, and Megan Hofmann. 2014. Coming to grips: 3D printing for accessibility. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility (ASSETS '14), 291--292.
[16]
Erin Buehler, Shaun K. Kane, and Amy Hurst. 2014. ABC and 3D: opportunities and obstacles to 3D printing in special education environments. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility (ASSETS '14), 107--114.
[17]
Giovanni Fusco and Valerie S. Morash. 2015. The Tactile Graphics Helper?: Providing Audio Clarification for Tactile Graphics Using Machine Vision Categories and Subject Descriptors. Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS '15): 491--500.
[18]
Timo Götzelmann. 2016. LucentMaps: 3D Printed Audiovisual Tactile Maps for Blind and Visually Impaired People. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16), 81--90.
[19]
Anhong Guo, Xiang "Anthony" Chen, Haoran Qi, Samuel White, Suman Ghosh, Chieko Asakawa, and Jeffrey P. Bigham. 2016. VizLens?: A Robust and Interactive Screen Reader for Interfaces in the Real World. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 651--664.
[20]
Anhong Guo, Jeeeun Kim, Xiang 'Anthony Chen, Tom Yeh, Scott E Hudson, Jennifer Mankoff, and Jeffrey P Bigham. 2017. Facade: Auto-generating Tactile Interfaces to Appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 5826--5838.
[21]
Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen. 2012. DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12), 389--402.
[22]
Mark Hatch. 2013. The Maker Movement Manifesto: Rules for Innovation in the New World of Crafters, Hackers, and Tinkerers. McGraw Hill Professional.
[23]
Megan Hofmann, Jeffrey Harris, Scott E. Hudson, and Jennifer Mankoff. 2016. Helping Hands: Requirements for a Prototyping Methodology for Upper-limb Prosthetics Users. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1769--1780.
[24]
David Huggins-daines, Mohit Kumar, Arthur Chan, Alan W Black, Mosur Ravishankar, and Alex I. Rudnicky. 2006. PocketSphinx: A free, real-time continuous speech recognition system for hand-held devices. In Proceedings of ICASSP.
[25]
Amy Hurst and Jasmine Tobias. 2011. Empowering individuals with do-it-yourself assistive technology. The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility (ASSETS '11): 11--18.
[26]
jaqtikkun. Cutaway Earth with Braille - Thingiverse. Retrieved April 1, 2017 from http://www.thingiverse.com/thing:510654
[27]
Shaun K. Kane and Jeffrey P. Bigham. 2014. Tracking @stemxcomet: teaching programming to blind students via 3D printing, crisis management, and twitter. In Proceedings of the 45th ACM technical symposium on Computer science education (SIGCSE '14), 247--252.
[28]
Shaun K. Kane, Brian Frey, and Jacob O. Wobbrock. 2013. Access Lens: A Gesture-Based Screen Reader for Real-World Documents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 347--350.
[29]
Jeeeun Kim and Tom Yeh. 2015. Toward 3D-Printed Movable Tactile Pictures for Children al Impairments with Visual Impairments. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2815--2824.
[30]
Michael A. Kolitsky. Making Tactile 3D Prints Talk. Retrieved April 2, 2017 from http://www.instructables.com/id/Making-Tactile-3D-Prints-Talk/?ALLSTEPS
[31]
Michael A. Kolitsky. 2015. 3D printing makes Visible Human cadaver sections accessible for blind students. Retrieved April 2, 2017 from http://www.nextgenemedia.com/NLM3Dprints/NLM3Dprints.html
[32]
Gierad Laput, Eric Brockmeyer, Scott E. Hudson, and Chris Harrison. 2015. Acoustruments: Passive, Acoustically-Driven, Interactive Controls for Handheld Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2161--2170.
[33]
Make: Assistive technology News, Reviews and More. Retrieved April 4, 2017 from http://makezine.com/tag/assistive-technology/
[34]
Assistive Technology Makers. Helping Makers Help Others. Retrieved April 4, 2017 from http://atmakers.org/
[35]
Samantha McDonald, Joshua Dutterer, Ali Abdolrahmani, Shaun K. Kane, and Amy Hurst. 2014. Tactile aids for visually impaired graphical design education. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility (ASSETS '14), 275--276.
[36]
Roderick Murray-Smith, John Williamson, Stephen Hughes, and Torben Quaade. 2008. Stane: Synthesized Surfaces for Tactile Input. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08): 1299--1302.
[37]
National Federation of the Blind Jernigan Institute. 2009. The Braille Literacy Crisis in America. National Federation of the Blind.
[38]
OSM Buildings. OSM Buildings. Retrieved April 4, 2017 from https://osmbuildings.org/
[39]
PaLEoS. Animal Cell - Thingiverse. Retrieved April 4, 2017 from http://www.thingiverse.com/thing:689381
[40]
Huaishu Peng, François Guimbretière, James Mccann, and Scott E Hudson. 2016. A 3D Printer for Interactive Electromagnetic Devices. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 553--562.
[41]
Huaishu Peng, Jennifer Mankoff, Scott E Hudson, and James McCann. 2015. A Layered Fabric 3D Printer for Soft Interactive Objects. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), 1789--1798.
[42]
Thomas Poon and Ronit Ovadia. 2008. Using Tactile Learning Aids for Students with Visual Impairments in a First-Semester Organic Chemistry Course. Journal of Chemical Education 85, 2: 240.
[43]
Andreas Reichinger, Anton Fuhrmann, Stefan Maierhofer, and Werner Purgathofer. 2016. Gesture-Based Interactive Audio Guide on Tactile Reliefs. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16), 91--100.
[44]
roman_hegglin. Customizable Atom Deluxe (every element preconfigured) - Thingiverse. Retrieved April 1, 2017 from http://www.thingiverse.com/thing:114247
[45]
Audrey C. Rule. 2011. Tactile Earth and Space Science Materials for Students with Visual Impairments: Contours, Craters, Asteroids, and Features of Mars. Journal of Geoscience Education 59, 4: 205--218.
[46]
Audrey C. Rule, Greg P. Stefanich, Robert M. Boody, and Belinda Peiffer. 2011. Impact of Adaptive Materials on Teachers and their Students with Visual Impairments in Secondary Science and Mathematics Classes. International Journal of Science Education 33, 6: 865--887.
[47]
Valkyrie Savage, Colin Chang, and Björn Hartmann. 2013. Sauron: embedded single-camera sensing of printed physical user interfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13), 447--456.
[48]
Valkyrie Savage, Sean Follmer, Jingyi Li, and Björn Hartmann. 2015. Makers' Marks: Physical Markup for Designing and Fabricating Functional Objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15), 103--108.
[49]
Valkyrie Savage, Andrew Head, Björn Hartmann, Dan B. Goldman, Gautham Mysore, and Wilmot Li. 2015. Lamello: Passive Acoustic Sensing for Tangible Input Components. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1277--1280.
[50]
Valkyrie Savage, Ryan Schmidt, Tovi Grossman, George Fitzmaurice, and Björn Hartmann. 2014. A series of tubes: adding interactivity to 3D prints using internal pipes. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14), 3--12.
[51]
Martin Schmitz, Mohammadreza Khalilbeigi, Matthias Balwierz, Roman Lissermann, Max Mühlhäuser, and Jürgen Steimle. 2015. Capricate: A Fabrication Pipeline to Design and 3D Print Capacitive Touch Sensors for Interactive Objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15), 253--258.
[52]
Huiying Shen, Owen Edwards, Joshua Miele, and James M. Coughlan. 2013. CamIO. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '13), 1--2.
[53]
Lei Shi. 2015. Talkabel: A Labeling Method for 3D Printed Models. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS '15), 361--362.
[54]
Lei Shi, Ross McLachlan, Yuhang Zhao, and Shiri Azenkot. 2016. Magic Touch: Interacting with 3D Printed Graphics. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16), 329--330.
[55]
Lei Shi, Idan Zelzer, Catherine Feng, and Shiri Azenkot. 2016. Tickers and Talker: An Accessible Labeling Toolkit for 3D Printed Models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4896--4907.
[56]
Lei Shi, Yuhang Zhao, and Shiri Azenkot. 2017. Designing Interactions for 3D Printed Models with Blind People. In In Proceedings of the 19th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS '17).
[57]
Roy Shilkrot, Jochen Huber, Wong Meng Ee, Pattie Maes, and Suranga Chandima Nanayakkara. 2015. FingerReader: A Wearable Device to Explore Printed Text on the Go. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2363--2372.
[58]
Hyunyoung Song, François Guimbretière, Chang Hu, and Hod Lipson. 2006. ModelCraft: capturing freehand annotations and edits on physical 3D models. In Proceedings of the 19th annual ACM symposium on User interface software and technology (UIST '06), 1--10.
[59]
Abigale Stangl, Jeeeun Kim, and Tom Yeh. 2014. 3D printed tactile picture books for children with visual impairments. In Proceedings of the 2014 conference on Interaction design and children (IDC '14), 321--324.
[60]
Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David A. Ross, and Jon E. Froehlich. 2016. Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras. ACM Transactions on Accessible Computing 9, 1: 1--38.
[61]
Saiganesh Swaminathan, Thijs Roumen, Robert Kovacs, David Stangl, Stefanie Mueller, and Patrick Baudisch. 2016. Linespace: A Sensemaking Platform for the Blind. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16): 2175--2185.
[62]
Brandon Taylor, Anind Dey, Dan Siewiorek, and Asim Smailagic. 2016. Customizable 3D Printed Tactile Maps as Interactive Overlays. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16), 71--79.
[63]
Jolene K. Teske, Phyllis Gray, Mason A. Kuhn, Courtney K. Clausen, Latisha L. Smith, Sukainah A. Alsubia, Maryam Ghayoorad, Audrey C. Rule, and Jean Suchsland Schneider. 2014. Teacher-made Tactile Science Materials with Critical and Creative Thinking Activities for Learners including those with Visual Impairments. Online Submission.
[64]
TOM. TOM: Tikkun Olam Makers. Retrieved April 4, 2017 from http://tomglobal.org/
[65]
Marynel Vázquez, Eric Brockmeyer, Ruta Desai, Chris Harrison, and Scott E. Hudson. 2015. 3D Printing Pneumatic Device Controls with Variable Activation Force Capabilities. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1295--1304.
[66]
Karl Willis, Eric Brockmeyer, Scott Hudson, and Ivan Poupyrev. 2012. Printed optics: 3D printing of embedded optical elements for interactive devices. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12), 589--598.
[67]
xehpuk. Simplified Tiger Tank - Thingiverse. Retrieved April 4, 2017 from http://www.thingiverse.com/thing:90265
[68]
Hansong Zhang and Kenneth E. Hoff. 1997. Fast backface culling using normal masks. In Proceedings of the 1997 symposium on Interactive 3D graphics (SI3D '97), 103-.
[69]
Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick?: Low - Cost Touch Sensing Using Electric Field Tomography. In In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 1--14.

Cited By

View all
  • (2024)Revisiting visual accessibility with non-textual content: challenges and solutions for human-computer interactionProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702105(1-18)Online publication date: 7-Oct-2024
  • (2024)Direct or Immersive? Comparing Smartphone-based Museum Guide Systems for Blind VisitorsProceedings of the 21st International Web for All Conference10.1145/3677846.3677856(10-22)Online publication date: 13-May-2024
  • (2024)Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual InteractionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676396(1-20)Online publication date: 13-Oct-2024
  • Show More Cited By

Index Terms

  1. Markit and Talkit: A Low-Barrier Toolkit to Augment 3D Printed Models with Audio Annotations

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology
    October 2017
    870 pages
    ISBN:9781450349819
    DOI:10.1145/3126594
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 October 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. 3D models
    2. computer vision
    3. visual impairments

    Qualifiers

    • Research-article

    Conference

    UIST '17

    Acceptance Rates

    UIST '17 Paper Acceptance Rate 73 of 324 submissions, 23%;
    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)56
    • Downloads (Last 6 weeks)10
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Revisiting visual accessibility with non-textual content: challenges and solutions for human-computer interactionProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702105(1-18)Online publication date: 7-Oct-2024
    • (2024)Direct or Immersive? Comparing Smartphone-based Museum Guide Systems for Blind VisitorsProceedings of the 21st International Web for All Conference10.1145/3677846.3677856(10-22)Online publication date: 13-May-2024
    • (2024)Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual InteractionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676396(1-20)Online publication date: 13-Oct-2024
    • (2024)Accessible Point-and-Tap Interaction for Acquiring Detailed Information About Tactile Graphics and 3D ModelsComputers Helping People with Special Needs10.1007/978-3-031-62846-7_30(252-259)Online publication date: 5-Jul-2024
    • (2024)The Museum of Touch: Tangible Models for Blind and Low Vision Audiences in MuseumsMultimodality in Architecture10.1007/978-3-031-49511-3_8(135-155)Online publication date: 13-Jan-2024
    • (2023)Optimization of 3D Immunofluorescence Analysis and Visualization Using IMARIS and MeshLabCells10.3390/cells1202021812:2(218)Online publication date: 4-Jan-2023
    • (2023)Understanding Experiences, Attitudes and Perspectives towards Designing Interactive Creative Tools for Teachers of Visually Impaired StudentsProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614512(1-5)Online publication date: 22-Oct-2023
    • (2023)Multimodal 3D printed urban maps for blind people. Evaluations and scientific investigationsProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614503(1-7)Online publication date: 22-Oct-2023
    • (2023)TouchPilot: Designing a Guidance System that Assists Blind People in Learning Complex 3D StructuresProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608426(1-18)Online publication date: 22-Oct-2023
    • (2023)Designing Conversational Multimodal 3D Printed Models with People who are BlindProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3595989(2172-2188)Online publication date: 10-Jul-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media