Mina Huh
Uran Oh
ABSTRACT
Webtoon is a type of digital comics read online where readers can leave comments to share their thoughts on the story. While it has experienced a surge in popularity internationally, people with visual impairments cannot enjoy webtoon with the lack of an accessible format. While traditional image description practices can be adopted, resulting descriptions cannot preserve webtoons’ unique values such as control over the reading pace and social engagement through comments. To improve the webtoon reading experience for BLV users, we propose Cocomix, an interactive webtoon reader that leverages comments into the design of novel webtoon interactions. Since comments can identify story highlights and provide additional context, we designed a system that provides 1) comments-based adaptive descriptions with selective access to details and 2) panel-anchored comments for easy access to relevant descriptive comments. Our evaluation (N=12) showed that Cocomix users could adapt the description for various needs and better utilize comments.
Supplemental Material
- Dragan Ahmetovic, Nahyun Kwon, Uran Oh, Cristian Bernareggi, and Sergio Mascetti. 2021. Touch Screen Exploration of Visual Artwork for Blind People. In Proceedings of the Web Conference 2021. 2781–2791.Google Scholar
Digital Library
- Kholoud Khalil Aldous, Jisun An, and Bernard J Jansen. 2019. View, like, comment, post: Analyzing user engagement by topic at 4 levels across 5 social media platforms for 53 news organizations. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 13. 47–57.Google ScholarCross Ref
- Chad Allen. [n.d.]. Unseen. Retrieved August 30, 2021 from https://www.unseencomic.com/Google Scholar
- Alphatart&Sumpul. [n.d.]. The Remarried Empress. Retrieved September 2, 2021 from https://www.webtoons.com/en/fantasy/the-remarried-empress/list?title_no=2135Google Scholar
- Alphatart/Sumpul. [n.d.]. The Remarried Empress. Retrieved August 30, 2021 from https://www.webtoons.com/en/fantasy/the-remarried-empress/list?title_no=2135Google Scholar
- Kohei Arai and Herman Tolle. 2010. Automatic e-comic content adaptation. International Journal of Ubiquitous Computing 1, 1 (2010), 1–11.Google Scholar
- Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2016. A simple but tough-to-beat baseline for sentence embeddings. (2016).Google Scholar
- artweb. [n.d.]. DESCRIVEDDO GUIDELINES. Retrieved September 9, 2021 from https://artweb.netlify.app/desc_enGoogle Scholar
- asimplebengo. [n.d.]. Hybrid. Retrieved September 2, 2021 from https://www.webtoons.com/en/challenge/hybrid/list?title_no=211861Google Scholar
- audiobook. [n.d.]. a movie in your mind. Retrieved August 30, 2021 from https://www.graphicaudiointernational.net/Google Scholar
- Olivier Augereau, Motoi Iwata, and Koichi Kise. 2018. A survey of comics research in computer science. Journal of imaging 4, 7 (2018), 87.Google ScholarCross Ref
- braille comics. [n.d.]. Laville Braille. Retrieved August 30, 2021 from http://www.lavillebraille.fr/des-livres-a-voir-et-a-toucher/Google Scholar
- João MC Correia and Abel JP Gomes. 2016. Balloon extraction from complex comic books using edge detection and histogram scoring. Multimedia Tools and Applications 75, 18 (2016), 11367–11390.Google ScholarDigital Library
- Jakob Dittmar. 2014. Comics for the blind and for the seeing. International Journal of Comic Art; 1 16 (2014).Google Scholar
- David Dubray and Jochen Laubrock. 2019. Deep CNN-based speech balloon detection and segmentation for comic books. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 1237–1243.Google ScholarCross Ref
- Arpita Dutta and Samit Biswas. 2019. CNN based extraction of panels/characters from bengali comic book page images. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), Vol. 1. IEEE, 38–43.Google ScholarCross Ref
- Teresa Kardoulias Sarah Stephenson Keyes Elisabeth Salzhauer Axel, Virginia Hooper and Francesca Rosenberg. [n.d.]. AEB’s Guidelines for Verbal Description. Retrieved September 9, 2021 from http://www.artbeyondsight.org/handbook/acs-guidelines.shtmlGoogle Scholar
- EmiMG. [n.d.]. ZomCom. Retrieved September 2, 2021 from https://www.webtoons.com/en/challenge/zomcom/list?title_no=70195Google Scholar
- Siamak Faridani, Ephrat Bitton, Kimiko Ryokai, and Ken Goldberg. 2010. Opinion space: a scalable tool for browsing online comments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1175–1184.Google ScholarDigital Library
- Anita Fidyka and Anna Matamala. 2018. Audio description in 360º videos. Translation Spaces (2018).Google Scholar
- Pet Foolery. [n.d.]. Pixie and Brutus. Retrieved September 2, 2021 from https://www.webtoons.com/en/challenge/pixie-and-brutus/list?title_no=452175Google Scholar
- Pet Foolery. [n.d.]. Pixie and Brutus. Retrieved August 30, 2021 from https://www.webtoons.com/en/challenge/pixie-and-brutus/list?title_no=452175Google Scholar
- Benjamin Fraser. 2020. Tactile comics, disability studies and the mind’s eye: on “A Boat Tour”(2017) in Venice with Max. Journal of Graphic Novels and Comics(2020), 1–13.Google Scholar
- Graham R Gibbs. 2007. Thematic coding and categorizing. Analyzing qualitative data 703 (2007), 38–56.Google ScholarCross Ref
- Cole Gleason, Amy Pavel, Himalini Gururaj, Kris Kitani, and Jeffrey P Bigham. 2020. Making GIFs Accessible.. In ASSETS. 24–1.Google Scholar
- Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia B Chilton, and Jeffrey P Bigham. 2019. Making memes accessible. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility. 367–376.Google ScholarDigital Library
- Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M Kitani, and Jeffrey P Bigham. 2020. Twitter A11y: A browser extension to make Twitter images accessible. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Matthew Honnibal and Mark Johnson. 2015. An Improved Non-monotonic Transition System for Dependency Parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, 1373–1378. https://aclweb.org/anthology/D/D15/D15-1162Google ScholarCross Ref
- Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Graber, Hal Daume, and Larry S Davis. 2017. The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition. 7186–7195.Google ScholarCross Ref
- Jakka. [n.d.]. Independant Diary. Retrieved August 30, 2021 from https://comic.naver.com/webtoon/list?titleId=748105&no=78&weekday=thuGoogle Scholar
- JH. [n.d.]. The Boxer. Retrieved August 30, 2021 from https://comic.naver.com/webtoon/list?titleId=736989&weekday=thuGoogle Scholar
- Violet Karim. [n.d.]. Familiar Feelings. Retrieved September 2, 2021 from https://www.webtoons.com/en/challenge/familiar-feelings/list?title_no=323558Google Scholar
- Hyunwoo Kim, Haesoo Kim, Kyung Je Jo, and Juho Kim. 2021. StarryThoughts: Facilitating Diverse Opinion Exploration on Social Issues. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1(2021), 1–29.Google ScholarDigital Library
- Jeffrey SJ Kirchoff. 2013. It’s just not the same as print (and it shouldn’t be): Rethinking the possibilities of digital comics. Technoculture: An Online Journal of Technology in Society 3, 1(2013).Google Scholar
- Soyoung Kwon and Kun-Pyo Lee. 2016. What makes readers laugh? value of sensing laughter for humor webtoon. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct. 867–874.Google Scholar
- Yunjung Lee, Hwayeon Joh, Suhyeon Yoo, and Uran Oh. 2021. AccessComics: an accessible digital comic book reader for people with visual impairments. In Proceedings of the 18th International Web for All Conference. 1–11.Google ScholarDigital Library
- Luyuan Li, Yongtao Wang, Liangcai Gao, Zhi Tang, and Ching Y Suen. 2014. Comic2CEBX: A system for automatic comic content adaptation. In IEEE/ACM Joint Conference on Digital Libraries. IEEE, 299–308.Google Scholar
- Guanhong Liu, Xianghua Ding, Chun Yu, Lan Gao, Xingyu Chi, and Yuanchun Shi. 2019. ” I Bought This for Me to Look More Ordinary” A Study of Blind People Doing Online Shopping. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–11.Google ScholarDigital Library
- Yang Liu. 2019. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318(2019).Google Scholar
- Zongyang Ma, Aixin Sun, Quan Yuan, and Gao Cong. 2012. Topic-driven reader comments summarization. In Proceedings of the 21st ACM international conference on Information and knowledge management. 265–274.Google ScholarDigital Library
- Scott McCloud. 1993. Understanding comics: The invisible art. Northampton, Mass (1993).Google Scholar
- Chris McCoy. [n.d.]. Safely. Retrieved September 2, 2021 from https://www.webtoons.com/en/comedy/safely-endangered/list?title_no=352Google Scholar
- Mimino666. [n.d.]. langdetect: Port of Google;s language-detection library. Retrieved September 9, 2021 from https://github.com/Mimino666/langdetect#languagesGoogle Scholar
- Elaheh Momeni, Claire Cardie, and Myle Ott. 2013. Properties, prediction, and prevalence of useful user-generated comments for descriptive annotation of social media objects. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 7.Google Scholar
- Elaheh Momeni, Ke Tao, Bernhard Haslhofer, and Geert-Jan Houben. 2013. Identification of useful user comments in social media: A case study on Flickr commons. In Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries. 1–10.Google ScholarDigital Library
- Valerie S Morash, Yue-Ting Siu, Joshua A Miele, Lucia Hasty, and Steven Landau. 2015. Guiding novice web workers in making image descriptions using templates. ACM Transactions on Accessible Computing (TACCESS) 7, 4 (2015), 1–21.Google ScholarDigital Library
- Giulio Mori, Maria Claudia Buzzi, Marina Buzzi, and Barbara Leporini. 2010. Structured audio podcasts via web text-to-speech system. In Proceedings of the 19th international conference on World wide web. 1281–1284.Google ScholarDigital Library
- Meredith Ringel Morris, Jazette Johnson, Cynthia L Bennett, and Edward Cutrell. 2018. Rich representations of visual content for screen reader users. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–11.Google ScholarDigital Library
- Murrz. [n.d.]. Murrz. Retrieved September 2, 2021 from https://www.webtoons.com/en/slice-of-life/murrz/list?title_no=1281Google Scholar
- Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity-level Factual Consistency of Abstractive Text Summarization. arXiv preprint arXiv:2102.09130(2021).Google Scholar
- Anime News Network. [n.d.]. Japanese Volunteers Transcribe Manga for Blind People. Retrieved August 30, 2021 from https://www.animenewsnetwork.com/news/2007-07-24/japanese-volunteers-transcribe-manga-for-blind-peopleGoogle Scholar
- Nhu-Van Nguyen, Christophe Rigaud, and Jean-Christophe Burie. 2017. Comic characters detection using deep learning. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), Vol. 3. IEEE, 41–46.Google ScholarCross Ref
- Toru Ogawa, Atsushi Otsubo, Rei Narita, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2018. Object detection for comics using manga109 annotations. arXiv preprint arXiv:1803.08670(2018).Google Scholar
- Pilar Orero, Stephen Doherty, Jan-Louis Kruger, Anna Matamala, Jan Pedersen, Elisa Perego, Pablo Romero-Fresco, Sara Rovira-Esteva, Olga Soler-Vilageliu, and Agnieszka Szarkowska. 2018. Conducting experimental research in audiovisual translation (AVT): A position paper. JosTrans: The Journal of Specialised Translation30 (2018), 105–126.Google Scholar
- Rachel Sarah Osolen and Leah Brochu. 2020. Creating an Authentic Experience. The International Journal of Information, Diversity, & Inclusion (IJIDI) 4, 1(2020).Google Scholar
- Xiaoran Qin, Yafeng Zhou, Yonggang Li, Siwei Wang, Yongtao Wang, and Zhi Tang. 2019. Progressive deep feature learning for manga character recognition via unlabeled training data. In Proceedings of the ACM Turing Celebration Conference-China. 1–6.Google ScholarDigital Library
- Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. arXiv preprint arXiv:1904.02594(2019).Google Scholar
- Frédéric Rayar. 2017. Accessible comics for visually impaired people: Challenges and opportunities. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 3. IEEE, 9–14.Google ScholarCross Ref
- Frédéric Rayar, Bernard Oriola, and Christophe Jouffrais. 2020. ALCOVE: an accessible comic reader for people with low vision. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 410–418.Google ScholarDigital Library
- Kyle Rector, Keith Salmon, Dan Thornton, Neel Joshi, and Meredith Ringel Morris. 2017. Eyes-free art: Exploring proxemic audio interfaces for blind and low vision art engagement. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1–21.Google ScholarDigital Library
- Kyle Reinholt, Darren Guinness, and Shaun K Kane. 2019. Eyedescribe: Combining eye gaze and speech to automatically create accessible touch screen artwork. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces. 101–112.Google ScholarDigital Library
- Christophe Rigaud. 2014. Segmentation and indexation of complex objects in comic book images. Ph.D. Dissertation. Université de La Rochelle.Google Scholar
- Christine Samson, Casey Fiesler, and Shaun K Kane. 2016. ” Holy Starches Batman!! We are Getting Walloped!” Crowdsourcing Comic Book Transcriptions. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility. 289–290.Google ScholarDigital Library
- Stefan Siersdorfer, Sergiu Chelaru, Wolfgang Nejdl, and Jose San Pedro. 2010. How useful are your comments? Analyzing and predicting YouTube comments and comment ratings. In Proceedings of the 19th international conference on World wide web. 891–900.Google ScholarDigital Library
- Rachel Smythe. [n.d.]. Lore Olympus. Retrieved September 2, 2021 from https://www.webtoons.com/en/romance/lore-olympus/list?title_no=1320Google Scholar
- Rachel Smythe. [n.d.]. Lore Olympus. Retrieved August 30, 2021 from https://www.webtoons.com/en/romance/lore-olympus/list?title_no=1320Google Scholar
- Abigale J Stangl, Esha Kothari, Suyog D Jain, Tom Yeh, Kristen Grauman, and Danna Gurari. 2018. Browsewithme: An online clothes shopping assistant for people with visual impairments. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. 107–118.Google ScholarDigital Library
- Raymond Hendy Susanto, Hai Leong Chieu, and Wei Lu. 2016. Learning to capitalize with character-level recurrent neural networks: an empirical study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2090–2095.Google ScholarCross Ref
- tactile comics. [n.d.]. Life - A tactile comic for blind people | Philipp Meyer. Retrieved August 30, 2021 from https://www.hallo.pm/life/Google Scholar
- Carla Tamburro, Timothy Neate, Abi Roper, and Stephanie Wilson. 2020. Accessible Creativity with a Comic Spin. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1–11.Google ScholarDigital Library
- Garreth W Tigwell, Benjamin M Gorman, and Rachel Menzies. 2020. Emoji Accessibility for Visually Impaired People. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Web Accessibility Tutorials. [n.d.]. Images Concepts - Images - WAI. Retrieved September 9, 2021 from https://www.graphicaudiointernational.net/Google Scholar
- uru chan. [n.d.]. unOrdinary. Retrieved September 2, 2021 from https://www.webtoons.com/en/super-hero/unordinary/episode-223/viewer?title_no=679&episode_no=234Google Scholar
- Violeta Voykinska, Shiri Azenkot, Shaomei Wu, and Gilly Leshed. 2016. How blind people interact with visual content on social networking services. In Proceedings of the 19th acm conference on computer-supported cooperative work & social computing. 1584–1595.Google ScholarDigital Library
- Ruolin Wang, Zixuan Chen, Mingrui Ray Zhang, Zhaoheng Li, Zhixiu Liu, Zihan Dang, Chun Yu, and Xiang’Anthony’ Chen. 2021. Revamp: Enhancing Accessible Information Seeking Experience of Online Shopping for Blind or Low Vision Users. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Xinwei Wang, Jun Hu, Bart Hengeveld, and Matthias Rauterberg. 2018. Segmentation of panels in d-Comics. In Interactivity, Game Creation, Design, Learning, and Innovation. Springer, 28–37.Google Scholar
- Xinwei Wang, Jun Hu, Bart Hengeveld, and Matthias Rauterberg. 2019. Expressing segmentation in d-comics. In International Conference on Human-Computer Interaction. Springer, 402–409.Google ScholarCross Ref
- Shaomei Wu and Lada A Adamic. 2014. Visually impaired users on an online social network. In Proceedings of the sigchi conference on human factors in computing systems. 3133–3142.Google ScholarDigital Library
- Shaomei Wu, Jeffrey Wieland, Omid Farivar, and Julie Schiller. 2017. Automatic alt-text: Computer-generated image descriptions for blind users on a social network service. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 1180–1192.Google ScholarDigital Library
- Yaongyi. [n.d.]. True Beauty. Retrieved September 2, 2021 from https://www.webtoons.com/en/romance/truebeauty/list?title_no=1436Google Scholar
- Matin Yarmand, Dongwook Yoon, Samuel Dodson, Ido Roll, and Sidney S Fels. 2019. ” Can you believe [1: 21]?!” Content and Time-Based Reference Patterns in Video Comments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential reasoning learning for language representation. arXiv preprint arXiv:2004.06870(2020).Google Scholar
- Jeffrey M Zacks, Barbara Tversky, and Gowri Iyer. 2001. Perceiving, remembering, and communicating structure in events.Journal of experimental psychology: General 130, 1 (2001), 29.Google Scholar
Index Terms
- Cocomix: Utilizing Comments to Improve Non-Visual Webtoon Accessibility
Recommendations
Comments-oriented blog summarization by sentence extraction
CIKM '07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge managementMuch existing research on blogs focused on posts only, ignoring their comments. Our user study conducted on summarizing blog posts, however, showed that reading comments does change one's understanding about blog posts. In this research, we aim to ...
The politics of comments: predicting political orientation of news stories with commenters' sentiment patterns
CSCW '11: Proceedings of the ACM 2011 conference on Computer supported cooperative workPolitical views frequently conflict in the coverage of contentious political issues, potentially causing serious social problems. We present a novel social annotation analysis approach for identification of news articles' political orientation. The ...
Comments-oriented document summarization: understanding documents with readers' feedback
SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrievalComments left by readers on Web documents contain valuable information that can be utilized in different information retrieval tasks including document search, visualization, and summarization. In this paper, we study the problem of comments-oriented ...
Comments