skip to main content
10.1145/3586183.3606784acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

TacNote: Tactile and Audio Note-Taking for Non-Visual Access

Published: 29 October 2023 Publication History

Abstract

Blind and visually impaired (BVI) people primarily rely on non-visual senses to interact with a physical environment.  Doing so requires a high cognitive load to perceive and memorize the presence of a large set of objects, such as at home or in a learning setting. In this work, we explored opportunities to enable object-centric note-taking  by using a 3D printing pen for interactive, personalized tactile annotations. We first identified the benefits and challenges of self-created tactile graphics in a formative diary study. Then, we developed TacNote, a system that enables BVI users to annotate, explore, and memorize critical information associated with everyday objects. Using TacNote, the users create tactile graphics with a 3D printing pen and attach them to the target objects. They capture and organize the physical labels  by using TacNote’s camera-based mobile app. In addition, they can specify locations, ordering, and hierarchy via finger-pointing interaction and receive audio feedback. Our user study with ten BVI participants showed that TacNote effectively alleviated the memory burden, offering a promising solution for enhancing users’ access to information.

Supplemental Material

ZIP File
Supplemental File

References

[1]
3Doodler. 2023. 3Doodler start+ [corporate products]. https://intl.the3doodler.com/collections/start.
[2]
Dustin Adams, Sri Kurniawan, Cynthia Herrera, Veronica Kang, and Natalie Friedman. 2016. Blind Photographers and VizSnap: A Long-Term Study. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (Reno, Nevada, USA) (ASSETS ’16). Association for Computing Machinery, New York, NY, USA, 201–208. https://doi.org/10.1145/2982142.2982169
[3]
Giovanni Anobile, Guido Marco Cicchini, and David C Burr. 2016. Number as a primary perceptual attribute: A review. Perception 45, 1-2 (2016), 5–31.
[4]
Arduino. 2023. Arduino Pro Mini [corporate products]. https://docs.arduino.cc/retired/boards/arduino-pro-mini.
[5]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3 (2009), 114–123.
[6]
Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh. 2010. VizWiz: Nearly Real-Time Answers to Visual Questions. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) (Raleigh, North Carolina) (W4A ’10). Association for Computing Machinery, New York, NY, USA, Article 24, 2 pages. https://doi.org/10.1145/1805986.1806020
[7]
Jens Bornschein, Denise Bornschein, and Gerhard Weber. 2018. Comparing computer-based drawing methods for blind people with real-time tactile feedback. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
[8]
Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel White, and Jeffrey P. Bigham. 2013. Visual Challenges in the Everyday Lives of Blind People. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2117–2126. https://doi.org/10.1145/2470654.2481291
[9]
John Brooke 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
[10]
Ruei-Che Chang, Wen-Ping Wang, Chi-Huan Chiang, Te-Yen Wu, Zheer Xu, Justin Luo, Bing-Yu Chen, and Xing-Dong Yang. 2021. AccessibleCircuits: Adaptive Add-On Circuit Components for People with Blindness or Low Vision. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
[11]
Cesare Cornoldi, Barbara Bertuccelli, Paola Rocchi, and Barbara Sbrana. 1993. Processing capacity limitations in pictorial and spatial representations in the totally congenitally blind. Cortex 29, 4 (1993), 675–689.
[12]
Steven C Dakin, Marc S Tibber, John A Greenwood, Frederick AA Kingdom, and Michael J Morgan. 2011. A common visual metric for approximate number and density. Proceedings of the National Academy of Sciences 108, 49 (2011), 19552–19557.
[13]
Josh Urban Davis, Te-Yen Wu, Bo Shi, Hanyi Lu, Athina Panotopoulou, Emily Whiting, and Xing-Dong Yang. 2020. TangibleCircuits: An interactive 3D printed circuit education tool for people with visual impairments. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
[14]
Nuzhah Gooda Sahib, Anastasios Tombros, and Tony Stockman. 2014. Investigating the behavior of visually impaired users for multi-session search tasks. Journal of the Association for Information Science and Technology 65, 1 (2014), 69–83.
[15]
Dimitrios B Goudiras, Konstantinos S Papadopoulos, Athanasios Ch Koutsoklenis, Virginia E Papageorgiou, and Maria S Stergiou. 2009. Factors affecting the reading media used by visually impaired adults. British Journal of Visual Impairment 27, 2 (2009), 111–127.
[16]
Anhong Guo, Xiang’Anthony’ Chen, Haoran Qi, Samuel White, Suman Ghosh, Chieko Asakawa, and Jeffrey P Bigham. 2016. Vizlens: A robust and interactive screen reader for interfaces in the real world. In Proceedings of the 29th annual symposium on user interface software and technology. 651–664.
[17]
Anhong Guo, Jeeeun Kim, Xiang’Anthony’ Chen, Tom Yeh, Scott E Hudson, Jennifer Mankoff, and Jeffrey P Bigham. 2017. Facade: Auto-generating tactile interfaces to appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5826–5838.
[18]
Anhong Guo, Junhan Kong, Michael Rivera, Frank F Xu, and Jeffrey P Bigham. 2019. Statelens: A reverse engineering solution for making existing dynamic touchscreens accessible. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 371–385.
[19]
Amina H. El-Ashry, Xinran Zhang, Savani Shrotri, Susanna Abler, and Foad Hamidi. 2021. Exploring the Collaboration Possibilities of Distributed Making for Storytelling Using 3D Printing Pens. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing (Virtual Event, USA) (CSCW ’21). Association for Computing Machinery, New York, NY, USA, 44–48. https://doi.org/10.1145/3462204.3481755
[20]
Hans Jørgen Wiberg. 2015. Be My Eyes [corporate Software]. https://www.bemyeyes.com/.
[21]
Liang He, Zijian Wan, Leah Findlater, and Jon E Froehlich. 2017. TacTILE: a preliminary toolchain for creating accessible graphics with 3D-printed overlays and auditory annotations. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. 397–398.
[22]
Megan Hofmann, Kelly Mack, Jessica Birchfield, Jerry Cao, Autumn G Hughes, Shriya Kurpad, Kathryn J Lum, Emily Warnock, Anat Caspi, Scott E Hudson, and Jennifer Mankoff. 2022. Maptimizer: Using Optimization to Tailor Tactile Maps to Users Needs. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 592, 15 pages. https://doi.org/10.1145/3491102.3517436
[23]
Leona Holloway, Matthew Butler, and Kim Marriott. 2022. 3D Printed Street Crossings: Supporting Orientation and Mobility Training with People who are Blind or have Low Vision. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–16.
[24]
Leona Holloway, Kim Marriott, and Matthew Butler. 2018. Accessible maps for the blind: Comparing 3D printed models with tactile graphics. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–13.
[25]
Chandrika Jayant, Hanjie Ji, Samuel White, and Jeffrey P. Bigham. 2011. Supporting Blind Photography. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (Dundee, Scotland, UK) (ASSETS ’11). Association for Computing Machinery, New York, NY, USA, 203–210. https://doi.org/10.1145/2049536.2049573
[26]
Tero Jokela, Jarno Ojala, and Thomas Olsson. 2015. A diary study on combining multiple information devices in everyday activities and tasks. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 3903–3912.
[27]
Vaishnav Kameswaran, Alexander J. Fiannaca, Melanie Kneisel, Amy Karlson, Edward Cutrell, and Meredith Ringel Morris. 2020. Understanding in-situ use of commonly available navigation technologies by people with visual impairments. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1–12.
[28]
Jeeeun Kim and Tom Yeh. 2015. Toward 3D-printed movable tactile pictures for children with visual impairments. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2815–2824.
[29]
Fabrizio Leo, Carla Tinti, Silvia Chiesa, Roberta Cavaglià, Susanna Schmidt, Elena Cocchi, and Luca Brayda. 2018. Improving spatial working memory in blind and sighted youngsters using programmable tactile displays. SAGE open medicine 6 (2018), 2050312118820028.
[30]
Jingyi Li, Son Kim, Joshua A Miele, Maneesh Agrawala, and Sean Follmer. 2019. Editing spatial layouts through tactile templates for people with visual impairments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–11.
[31]
Jiasheng Li, Zeyu Yan, Ebrima Haddy Jarjue, Ashrith Shetty, and Huaishu Peng. 2022. TangibleGrid: Tangible Web Layout Design for Blind Users. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 47, 12 pages. https://doi.org/10.1145/3526113.3545627
[32]
David G Lowe. 1999. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, Vol. 2. Ieee, 1150–1157.
[33]
David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60 (2004), 91–110.
[34]
Magnifying Aids. 2023. Braille Labeler [corporate products]. https://www.magnifyingaids.com/Braille_Labeler, https://www.magnifyingaids.com/6dot-Braille-Labeler.
[35]
David McGookin, Euan Robertson, and Stephen Brewster. 2010. Clutching at Straws: Using Tangible Interaction to Provide Non-Visual Access to Graphs. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 1715–1724. https://doi.org/10.1145/1753326.1753583
[36]
OpenCV Open Source Computer Vision. -. Morphological Transformations [Software]. https://docs.opencv.org/4.x/d9/d61/tutorial_py_morphological_ops.html.
[37]
OpenCV Open Source Computer Vision. -. Structural Analysis and Shape Descriptors [Software]. https://docs.opencv.org/3.4/d3/dc0/group__imgproc__shape.html.
[38]
Maulishree Pandey, Hariharan Subramonyam, Brooke Sasia, Steve Oney, and Sile O’Modhrain. 2020. Explore, create, annotate: designing digital drawing tools with visually impaired people. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–12.
[39]
Noa Raz, Ella Striem, Golan Pundak, Tanya Orlov, and Ehud Zohary. 2007. Superior serial memory in the blind: a case of cognitive compensatory adjustment. Current Biology 17, 13 (2007), 1129–1133. https://doi.org/10.1016/j.cub.2007.05.060
[40]
Kathryn Ringland. 2013. Accessible Clothing Tags: Designing for Individuals with Visual Impairments. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 2749–2754. https://doi.org/10.1145/2468356.2479504
[41]
Thijs Roumen, Bastian Kruck, Tobias Dürschmid, Tobias Nack, and Patrick Baudisch. 2016. Mobile fabrication. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 3–14.
[42]
Saiph Savage, Claudia Flores-Saviaga, Rachel Rodney, Liliana Savage, Jon Schull, and Jennifer Mankoff. 2022. The Global Care Ecosystems of 3D Printed Assistive Devices. ACM Trans. Access. Comput. 15, 4, Article 31 (oct 2022), 29 pages. https://doi.org/10.1145/3537676
[43]
Muhammad Ikmal Hakim Shamsul Bahrin, Hazlina Md Yusof, and Shahrul Na’im Sidek. 2022. Hands and fingers tracking for tactile graphics reading assistive device. In Enabling Industry 4.0 through Advances in Mechatronics: Selected Articles from iM3F 2021, Malaysia. Springer, 413–422.
[44]
Rita Shewbridge, Amy Hurst, and Shaun K Kane. 2014. Everyday making: identifying future uses for 3D printing in the home. In Proceedings of the 2014 conference on Designing interactive systems. 815–824.
[45]
Lei Shi, Yuhang Zhao, Ricardo Gonzalez Penuela, Elizabeth Kupferstein, and Shiri Azenkot. 2020. Molder: An Accessible Design Tool for Tactile Maps. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376431
[46]
Alexa F. Siu, Son Kim, Joshua A. Miele, and Sean Follmer. 2019. ShapeCAD: An Accessible 3D Modelling Workflow for the Blind and Visually-Impaired Via 2.5D Shape Displays. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 342–354. https://doi.org/10.1145/3308561.3353782
[47]
Haruki Takahashi and Jeeeun Kim. 2019. 3D pen+ 3D printer: Exploring the role of humans and fabrication machines in creative making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
[48]
Lauren Thévin, Christophe Jouffrais, Nicolas Rodier, Nicolas Palard, Martin Hachet, and Anke M Brock. 2019. Creating accessible interactive audio-tactile drawings using spatial augmented reality. In Proceedings of the 2019 ACM international conference on interactive surfaces and spaces. 17–28.
[49]
Lauren Thevin, Nicolas Rodier, Bernard Oriola, Martin Hachet, Christophe Jouffrais, and Anke M Brock. 2021. Inclusive adaptation of existing board games for gamers with and without visual impairments using a spatial augmented reality framework for touch detection and audio feedback. Proceedings of the ACM on Human-Computer Interaction 5, ISS (2021), 1–33.
[50]
Karen A Toussaint and Jeffrey H Tiger. 2010. Teaching early braille literacy skills within a stimulus equivalence paradigm to children with degenerative visual impairments. Journal of applied behavior analysis 43, 2 (2010), 181–194.
[51]
Andrey Vakunov, Chuo-Ling Chang, Fan Zhang, George Sung, Matthias Grundmann, and Valentin Bazarevsky. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. https://mixedreality.cs.cornell.edu/workshop.
[52]
Robert W Van Boven, Roy H Hamilton, Thomas Kauffman, Julian P Keenan, and Alvaro Pascual-Leone. 2000. Tactile spatial resolution in blind Braille readers. Neurology 54, 12 (2000), 2230–2236.
[53]
Tomaso Vecchi, Carla Tinti, and Cesare Cornoldi. 2004. Spatial memory and integration processes in congenital blindness. Neuroreport 15, 18 (2004), 2787–2790.
[54]
Xiyue Wang, Seita Kayukawa, Hironobu Takagi, and Chieko Asakawa. 2022. BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–14.
[55]
Chih-Fu Wu, Hsiang-Ping Wu, Yung-Hsiang Tu, and I-Ting Yeh. 2020. 3D pen tactile pictures generated by individuals with visual impairments. Journal of Visual Impairment & Blindness 114, 5 (2020), 382–392.
[56]
Maralbek Zeinullin and Marion Hersh. 2022. Tactile Audio Responsive Intelligent System. IEEE Access 10 (2022), 122074–122091. https://doi.org/10.1109/ACCESS.2022.3223099
[57]
Xiaoyi Zhang, Tracy Tran, Yuqian Sun, Ian Culhane, Shobhit Jain, James Fogarty, and Jennifer Mankoff. 2018. Interactiles: 3D printed tactile interfaces to enhance mobile touchscreen accessibility. In Proceedings of the 20th international ACM SIGACCESS conference on computers and accessibility. 131–142.
[58]
Yu Zhong, Pierre J. Garrigues, and Jeffrey P. Bigham. 2013. Real Time Object Scanning Using a Mobile Phone and Cloud-Based Visual Search Engine. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (Bellevue, Washington) (ASSETS ’13). Association for Computing Machinery, New York, NY, USA, Article 20, 8 pages. https://doi.org/10.1145/2513383.2513443
[59]
Yu Zhong, Walter S. Lasecki, Erin Brady, and Jeffrey P. Bigham. 2015. RegionSpeak: Quick Comprehensive Spatial Descriptions of Complex Images for Blind Users. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 2353–2362. https://doi.org/10.1145/2702123.2702437

Cited By

View all
  • (2025)How Can Haptic Feedback Assist People with Blind and Low Vision (BLV): A Systematic Literature ReviewACM Transactions on Accessible Computing10.1145/371193118:1(1-57)Online publication date: 13-Jan-2025
  • (2025)Haptic Interaction Methods for Freehand Contour Generation on a Refreshable Pin DisplayJournal of Computing and Information Science in Engineering10.1115/1.406741725:3Online publication date: 27-Jan-2025
  • (2024)Assessment in the Digital AgePreparing Students for the Future Educational Paradigm10.4018/979-8-3693-1536-1.ch003(47-69)Online publication date: 15-Mar-2024

Index Terms

  1. TacNote: Tactile and Audio Note-Taking for Non-Visual Access

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
      October 2023
      1825 pages
      ISBN:9798400701320
      DOI:10.1145/3586183
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 29 October 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. 3D printing pen
      2. Accessibility
      3. Assistive technology
      4. Fabrication
      5. Mobile Application
      6. Tactile graphics

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      • National Science and Technology Council of Taiwan

      Conference

      UIST '23

      Acceptance Rates

      Overall Acceptance Rate 561 of 2,567 submissions, 22%

      Upcoming Conference

      UIST '25
      The 38th Annual ACM Symposium on User Interface Software and Technology
      September 28 - October 1, 2025
      Busan , Republic of Korea

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)211
      • Downloads (Last 6 weeks)15
      Reflects downloads up to 11 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)How Can Haptic Feedback Assist People with Blind and Low Vision (BLV): A Systematic Literature ReviewACM Transactions on Accessible Computing10.1145/371193118:1(1-57)Online publication date: 13-Jan-2025
      • (2025)Haptic Interaction Methods for Freehand Contour Generation on a Refreshable Pin DisplayJournal of Computing and Information Science in Engineering10.1115/1.406741725:3Online publication date: 27-Jan-2025
      • (2024)Assessment in the Digital AgePreparing Students for the Future Educational Paradigm10.4018/979-8-3693-1536-1.ch003(47-69)Online publication date: 15-Mar-2024

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media