skip to main content
research-article

Colorblind-shareable videos by synthesizing temporal-coherent polynomial coefficients

Published: 08 November 2019 Publication History

Abstract

To share the same visual content between color vision deficiencies (CVD) and normal-vision people, attempts have been made to allocate the two visual experiences of a binocular display (wearing and not wearing glasses) to CVD and normal-vision audiences. However, existing approaches only work for still images. Although state-of-the-art temporal filtering techniques can be applied to smooth the per-frame generated content, they may fail to maintain the multiple binocular constraints needed in our applications, and even worse, sometimes introduce color inconsistency (same color regions map to different colors). In this paper, we propose to train a neural network to predict the temporal coherent polynomial coefficients in the domain of global color decomposition. This indirect formulation solves the color inconsistency problem. Our key challenge is to design a neural network to predict the temporal coherent coefficients, while maintaining all required binocular constraints. Our method is evaluated on various videos and all metrics confirm that it outperforms all existing solutions.

Supplementary Material

ZIP File (a174-xinghong.zip)
Supplemental files.

References

[1]
2015. Spectral Edge for displays. https://www.spectraledge.co.uk/spectral-edge-for-displays7. Online; accessed 26-April-2018.
[2]
Tunç Ozan Aydin, Nikolce Stefanoski, Simone Croci, Markus Gross, and Aljoscha Smolic. 2014. Temporally Coherent Local Tone Mapping of HDR Video. ACM Trans. Graph. 33, 6, Article 196 (Nov. 2014), 13 pages.
[3]
Nicolas Bonneel, Kalyan Sunkavalli, Sylvain Paris, and Hanspeter Pfister. 2013. Example-based Video Color Grading. ACM Trans. Graph. 32, 4, Article 39 (July 2013), 12 pages.
[4]
Nicolas Bonneel, Kalyan Sunkavalli, James Tompkin, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2014. Interactive Intrinsic Video Editing. ACM Trans. Graph. 33, 6, Article 197 (Nov. 2014), 10 pages.
[5]
Nicolas Bonneel, James Tompkin, Deqing Sun, Oliver Wang, Kalyan Sunkavalli, Sylvain Paris, and Hanspeter Pfister. 2017. Consistent video filtering for camera arrays. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 397--407.
[6]
Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015. Blind Video Temporal Consistency. ACM Trans. Graph. 34, 6, Article 196 (Oct. 2015), 9 pages.
[7]
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017. Coherent online video style transfer. In Proc. Intl. Conf. Computer Vision (ICCV).
[8]
Soon Hau Chua, Haimo Zhang, Muhammad Hammad, Shengdong Zhao, Sahil Goyal, and Karan Singh. 2015. ColorBless: Augmenting Visual Information for Colorblind People with Binocular Luster Effect. ACM Trans. Comput.-Hum. Interact. 21, 6, Article 32 (Jan. 2015), 20 pages.
[9]
Eugenio Culurciello and Alfredo Canziani. 2017. e-Lab Video Data Set. https://engineering.purdue.edu/elab/eVDS/.
[10]
Qingnan Fan, Jiaolong Yang, David P. Wipf, Baoquan Chen, and Xin Tong. 2018. Image Smoothing via Unsupervised Learning. CoRR abs/1811.02804 (2018). arXiv:1811.02804 http://arxiv.org/abs/1811.02804
[11]
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249--256.
[12]
Xinghong Hu, Zhuming Zhang, Xueting Liu, and Tien-Tsin Wong. 2019. Deep Visual Sharing with Colorblind. IEEE Transactions on Computational Imaging (2019).
[13]
Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin. 2018. Exposure: A White-Box Photo Post-Processing Framework. ACM Transactions on Graphics (TOG) 37, 2 (2018), 26.
[14]
Chun-Rong Huang, Kuo-Chuan Chiu, and Chu-Song Chen. 2011. Temporal color consistency-based video reproduction for dichromats. IEEE Transactions on Multimedia 13, 5 (2011), 950--960.
[15]
Jia-Bin Huang, Yu-Cheng Tseng, Se-In Wu, and Sheng-Jyh Wang. 2007. Information preserving color transformation for protanopia and deuteranopia. Signal Processing Letters, IEEE 14, 10 (2007), 711--714.
[16]
Mark J Huiskes and Michael S Lew. 2008. The MIR flickr retrieval evaluation. In Proceedings of the 1st ACM international conference on Multimedia information retrieval. ACM, 39--43.
[17]
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2017. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1647--1655.
[18]
Bernd Jähne, Horst Haussecker, and Peter Geissler. 1999. Handbook of computer vision and applications. Vol. 2. Citeseer.
[19]
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. 2017. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. arXiv preprint arXiv:1712.00080 (2017).
[20]
Tony Johnson. 1996. Methods for characterizing colour scanners and digital cameras. Displays 16, 4 (1996), 183--191.
[21]
D Kinga and J Ba Adam. 2015. A method for stochastic optimization. In International Conference on Learning Representations (ICLR), Vol. 5.
[22]
Naejin Kong, Peter V. Gehler, and Michael J. Black. 2014. Intrinsic Video. In Computer Vision - ECCV 2014 (Lecture Notes in Computer Science), Vol. 8690. Springer International Publishing, 360--375.
[23]
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. 2018. Learning blind video temporal consistency. In Proceedings of the European Conference on Computer Vision (ECCV). 170--185.
[24]
Manuel Lang, Oliver Wang, Tunc Aydin, Aljoscha Smolic, and Markus Gross. 2012. Practical Temporal Consistency for Image-based Graphics Applications. ACM Trans. Graph. 31, 4, Article 34 (July 2012), 8 pages.
[25]
Cewu Lu, Li Xu, and Jiaya Jia. 2012. Contrast preserving decolorization. In Computational Photography (ICCP), 2012 IEEE International Conference on. IEEE, 1--7.
[26]
Gustavo M Machado and Manuel M Oliveira. 2010. Real-Time Temporal-Coherent Color Contrast Enhancement for Dichromats. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 933--942.
[27]
Gustavo M Machado, Manuel M Oliveira, and Leandro AF Fernandes. 2009. A physiologically-based model for simulation of color vision deficiency. IEEE Transactions on Visualization and Computer Graphics 15, 6 (2009), 1291--1298.
[28]
Sylvain Paris. 2008. Edge-Preserving Smoothing and Mean-Shift Segmentation of Video Streams. Springer Berlin Heidelberg, Berlin, Heidelberg, 460--473.
[29]
Behzad Sajadi, Aditi Majumder, Manuel M Oliveira, Rosália G Schneider, and Ramesh Raskar. 2013. Using patterns to encode color information for dichromats. IEEE transactions on visualization and computer graphics 19, 1 (2013), 118--129.
[30]
Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-Recurrent Video Super-Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6626--6634.
[31]
Wuyao Shen, Xiangyu Mao, Xinghong Hu, and Tien-Tsin Wong. 2016. Seamless Visual Sharing with Color Vision Deficiencies. ACM Trans. Graph. 35, 4, Article 70 (July 2016), 12 pages.
[32]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012).
[33]
Chung-Ming Wang, Yao-Hsien Huang, and Ming-Long Huang. 2006. An effective algorithm for image sequence color transfer. Mathematical and Computer Modelling 44, 7--8 (2006), 608--627.
[34]
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. Video-to-video synthesis. arXiv preprint arXiv:1808.06601 (2018).
[35]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.
[36]
Stephen Wolf. 2003. Color correction matrix for digital still and video imaging systems. National Telecommunications and Information Administration Washington, DC.
[37]
Xuan Yang, Linling Zhang, Tien-Tsin Wong, and Pheng-Ann Heng. 2012. Binocular Tone Mapping. ACM Transactions on Graphics 31, 4 (2012), 93:1--93:10.
[38]
Genzhi Ye, Elena Garces, Yebin Liu, Qionghai Dai, and Diego Gutierrez. 2014. Intrinsic Video and Applications. ACM Trans. Graph. 33, 4, Article 80 (July 2014), 11 pages.
[39]
Zhuming Zhang, Chu Han, Shengfeng He, Xueting Liu, Haichao Zhu, Xinghong Hu, and Tien-Tsin Wong. 2019. Deep binocular tone mapping. The Visual Computer (2019), 1--15. https://link.springer.com/article/10.1007/s00371-019-01669-8
[40]
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40, 6 (2018), 1452--1464.

Cited By

View all
  • (2024)Building Coarse to Fine Convex Hulls With Auxiliary Vertices for Palette-Based Image RecoloringIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329638630:8(5581-5595)Online publication date: Aug-2024
  • (2024)Perceptual Uniformity-Aware Image Recoloring Method for Red-Green Anomalous Trichromacy2024 Nicograph International (NicoInt)10.1109/NICOInt62634.2024.00017(41-48)Online publication date: 14-Jun-2024
  • (2024)Image recoloring for color vision deficiency compensation using Swin transformerNeural Computing and Applications10.1007/s00521-023-09367-236:11(6051-6066)Online publication date: 1-Apr-2024
  • Show More Cited By

Index Terms

  1. Colorblind-shareable videos by synthesizing temporal-coherent polynomial coefficients

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Graphics
    ACM Transactions on Graphics  Volume 38, Issue 6
    December 2019
    1292 pages
    ISSN:0730-0301
    EISSN:1557-7368
    DOI:10.1145/3355089
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 November 2019
    Published in TOG Volume 38, Issue 6

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. color vision deficiency
    2. machine learning
    3. temporal coherence

    Qualifiers

    • Research-article

    Funding Sources

    • Hong Kong Research Grants Council

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)11
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Building Coarse to Fine Convex Hulls With Auxiliary Vertices for Palette-Based Image RecoloringIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329638630:8(5581-5595)Online publication date: Aug-2024
    • (2024)Perceptual Uniformity-Aware Image Recoloring Method for Red-Green Anomalous Trichromacy2024 Nicograph International (NicoInt)10.1109/NICOInt62634.2024.00017(41-48)Online publication date: 14-Jun-2024
    • (2024)Image recoloring for color vision deficiency compensation using Swin transformerNeural Computing and Applications10.1007/s00521-023-09367-236:11(6051-6066)Online publication date: 1-Apr-2024
    • (2024)Fast image recoloring for red–green anomalous trichromacy with contrast enhancement and naturalness preservationThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-024-03454-840:7(4647-4660)Online publication date: 1-Jul-2024
    • (2023)CC-Glasses: Color Communication Support for People with Color Vision Deficiency Using Augmented Reality and Deep LearningProceedings of the Augmented Humans International Conference 202310.1145/3582700.3582707(190-199)Online publication date: 12-Mar-2023
    • (2023)Artificial intelligence for visually impairedDisplays10.1016/j.displa.2023.10239177(102391)Online publication date: Apr-2023
    • (2022)Personalized Image Recoloring for Color Vision Deficiency CompensationIEEE Transactions on Multimedia10.1109/TMM.2021.307010824(1721-1734)Online publication date: 2022

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media