research-article

Mixed-primary factorization for dual-frame computational displays

Authors:

Fu-Chung Huang,

David LuebkeAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 36, Issue 4

Article No.: 149, Pages 1 - 13

https://doi.org/10.1145/3072959.3073654

Published: 20 July 2017 Publication History

Abstract

Increasing resolution and dynamic range of digital color displays is challenging with designs confined by cost and power specifications. This necessitates modern displays to trade-off spatial and temporal resolution for color reproduction capability. In this work we explore the idea of joint hardware and algorithm design to balance such trade-offs. We introduce a system that uses content-adaptive and compressive factorizations to reproduce colors. Each target frame is factorized into two products of high-resolution monochromatic and low-resolution color images, which then get integrated through temporal or spatial multiplexing. As our framework minimizes the error in colorimetric space, the perceived color rendition is high, and thanks to GPU acceleration, the results are generated in real-time. We evaluate our system with a LCD prototype that uses LED backlight array and temporal multiplexing to reproduce color images. Our approach enables high effective resolution and dynamic range without increasing power consumption. We also demonstrate low-cost extensions to hyperspectral and light-field imaging, which are possible due to compressive nature of our system.

Supplementary Material

ZIP File (a149-huang.zip)

Supplemental files.

Download
511.29 MB

MP4 File (papers-0318.mp4)

Download
474.17 MB

References

[1]

Michael Abrash. 2013. Why virtual isn't real to your brain: judder. http://blogs.valvesoftware.com/abrash/why-virtual-isnt-real-to-your-brain-judder/. (2013).

[2]

Takeyuki Ajito, Takashi Obi, Masahiro Yamaguchi, and Nagaaki Ohyama. 2000. Expanded color gamut reproduced by six-primary projection display. Proc. SPIE 3954, 130--137.

[3]

C Baumann. 1992. Ewald Hering's Opponent Colors. History of an idea. Der Ophthalmologe : Zeitschrift der Deutschen Ophthalmologischen Gesellschaft 89, 3 (1992), 249--252.

[4]

Bryce E. Bayer. 1976. Color imaging array. U.S. Patent 3971065 A. (1976).

[5]

Johan Bergquist and Carl Wennstam. 2006. Field-Sequential-Colour Display with Adaptive Gamut. SID Symposium Digest of Technical Papers 37, 1 (2006), 1594--1597.

[6]

Michael W. Berry, Murray Browne, Amy N. Langville, V. Paul Pauca, and Robert J. Plemmons. 2006. Algorithms and Applications for Approximate Nonnegative Matrix Factorization. In Computational Statistics and Data Analysis. 155--173.

[7]

Floraine Berthouzoz and Raanan Fattal. 2012. Resolution Enhancement by Vibrating Displays. ACM Trans. Graph. 31, 2, Article 15 (2012).

Digital Library

[8]

V. Blondel, N-D Ho, and P van Dooren. 2008. Weighted Nonnegative Matrix Factorization and Face Feature Extraction. In Image and Vision Computing. 1--17.

[9]

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. 2011. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Found. Trends Mach. Learn. 3, 1 (2011), 1--122.

Digital Library

[10]

Bruce Bridgeman, Derek Hendry, and Lawrence Stark. 1975. Failure to detect displacement of the visual world during saccadic eye movements. Vision Research 15, 6 (1975), 719 -- 722.

[11]

Haidong Chen, Ji Wang, Weifeng Chen, Huamin Qu, and Wei Chen. 2014. An image-space energy-saving visualization scheme for OLED displays. Computers & Graphics 38 (2014), 61 -- 68.

Digital Library

[12]

Weifeng Chen, Wei Chen, Haidong Chen, Zhengfang Zhang, and Huamin Qu. 2016. An Energy-saving Color Scheme for Direct Volume Rendering. Comput. Graph. 54, C (Feb. 2016), 57--64.

Digital Library

[13]

Wei-Chung Cheng,Yu Hou, and Massoud Pedram. 2004. Power minimization in a backlit TFT-LCD display by concurrent brightness and contrast scaling. In Proceedings Design, Automation and Test in Europe Conference and Exhibition, Vol. 1. 252--257 Vol.1.

[14]

Yu-Kuo Cheng, Yi-Pai Huang, Yi-Ru Cheng, and Han-Ping D. Shieh. 2009. Two-Field Scheme: Spatiotemporal Modulation for Field Sequential Color LCDs. Journal of Display Technology 5, 10 (2009).

[15]

Johnson Chuang, Daniel Weiskopf, and Torsten Möller. 2009. Energy Aware Color Sets. Computer Graphics Forum 28, 2 (2009), 203--211.

[16]

Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Apparent Display Resolution Enhancement for Moving Images (SIGGRAPH 2010). ACM, Article 113, 8 pages.

[17]

Mark Fairchild. 2013. Color appearance models. John Wiley & Sons.

[18]

Mark D. Fairchild and Ping-Hsu Chen. 2011. Brightness, lightness, and specifying color in high-dynamic-range scenes and images. Proc. SPIE 7867.

[19]

Mark D. Fairchild and Garrett M. Johnson. 2004. The iCAM framework for image appearance, image differences, and image quality. Journal of Electronic Imaging 13 (2004), 126--138.

[20]

Raanan Fattal. 2014. Dehazing using Color-Lines. ACM Transaction on Graphics 34, 1 (2014), 13:1--13:14.

Digital Library

[21]

James A. Ferwerda, Peter Shirley, Sumanta N. Pattanaik, and Donald P. Greenberg. 1997. A Model of Visual Masking for Computer Graphics. In SIGGRAPH. 143--152.

Digital Library

[22]

N. Fisekovic, T. Nauta, H. Cornelissen, and J. Bruinink. 2001. Improved motion-picture quality of AM-LCDs using scanning backlight. In Asia Display/IDW. 1637--1640.

[23]

Stephen R. Forrest. 2003. The road to high efficiency organic light emitting devices. Organic Electronics 4, 2--3 (2003), 45 -- 48.

[24]

A. Frankenstein and Werner von Jaworski. 1904. Verfahren und Vorrichtung zur Fernsichtbarmachung von Bildern und Gegenständen mittels Selenzellen, Dreifarbenfilter, und Zerlegung des Bildes in Punktgruppen durch Spiegelung. German Patent 172376. (1904).

[25]

Felix Heide, Douglas Lanman, Dikpal Reddy, Jan Kautz, Kari Pulli, and David Luebke. 2014. Cascaded Displays: Spatiotemporal Superresolution Using Offset Pixel Layers. ACM Trans. Graph. (SIGGRAPH) 33, 4 (2014), 60:1--60:11.

Digital Library

[26]

K.I. Iourcha, K.S. Nayak, and Z. Hong. 1999. System and method for fixed-rate block-based image compression with inferred pixel values. US Patent 5,956,431. (1999).

[27]

Garrett M. Johnson, Xioyan Song, Ethan D. Montag, and Mark D. Fairchild. 2010. Derivation of a Color Space for Image Color Difference Measurement. Color Research and Application 35, 6 (2010).

[28]

Kälil Käläntär, Tadashi Kishimoto, Kazuo Sekiya, Tetsuya Miyashita, and Tatsuo Uchida. 2006. Spatio-temporal scanning backlight mode for field-sequential-color optically-compensated-bend liquid-crystal display. Journal of SID 14, 2 (2006).

[29]

I. Kauvar, S. Yang, L. Shi, I. McDowall, and G. Wetzstein. 2015. Adaptive Color Display via Perceptually-driven Factored Spectral Projection. ACM Trans. Graph. (SIGGRAPH Asia) 34, 6 (2015).

Digital Library

[30]

Gorham Kindem. 1981. The Demise of Kinemacolor: Technological, Legal, Economic, and Aesthetic Problems in Early Color Cinema History. Cinema Journal 20, 2 (1981), 3--14.

[31]

Michiel A. Klompenhouwer. 2006. Comparison of LCD Motion Blur Reduction Methods using Temporal Impulse Response and MPRT. SID Symposium Digest of Technical Papers 37, 1 (2006).

[32]

Edwin H. Land. 1959. Experiments in Color Vision. Scientific American 200, 5 (1959), 84--94.

[33]

Erno H.A. Langendijk. 2007. A novel spectrum-sequential display design with a wide color gamut and reduced color breakup. Journal of the Society for Information Display 15, 4 (2007), 261--266.

[34]

Y. H. Liu, Z. Z. Yang, and S. C. Wang. 2010. A novel sequential-color RGB-LED backlight driving system with local dimming control and dynamic bus voltage regulation. IEEE Transactions on Consumer Electronics 56, 4 (2010).

Digital Library

[35]

Dhruv Mahajan, Ira Kemelmacher Shlizerman, Ravi Ramamoorthi, and Peter Belhumeur. 2007. A Theory of Locally Low Dimensional Light Transport. ACM Transaction on Graphics (SIGGRAPH) 26, 3 (2007), 62:1--62:10.

[36]

Rafal Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM Trans. Graph. 30, 4 (2011).

Digital Library

[37]

Susana Martinez-Conde, Jorge Otero-Millan, and Stephen L Macknik. 2013. The impact of microsaccades on vision: towards a unified theory of saccadic function. Nature reviews. Neuroscience (2013).

[38]

Belen Masia, Gordon Wetzstein, Piotr Didyk, and Diego Gutierrez. 2013. A survey on computational displays: Pushing the boundaries of optics, computation, and perception. Computers & Graphics 37, 8 (2013), 1012 -- 1038.

Digital Library

[39]

David Melcher. 2011. Visual stability. Philosophical Transactions of the Royal Society of London B: Biological Sciences 366, 1564 (2011), 468--475.

[40]

Ankit Mohan, Ramesh Raskar, and Jack Tumblin. 2008. Agile Spectrum Imaging: Programmable Wavelength Modulation for Cameras and Projectors. Computer Graphics Forum 27, 2 (2008), 709--717.

[41]

Mineo Mori, Toyohiko Hatada, Kazuo Ishikawa, Tosio Saishouji, Osamu Wada, Junichi Nakamura, and Nobuyoshi Terashima. 1999. Mechanism of color breakup in field-sequential-color projectors. Journal of the Society for Information Display 7, 4 (1999), 257--259.

[42]

V. G. Moshnyaga and E. Morikawa. 2005. LCD display energy reduction by user monitoring. In 2005 International Conference on Computer Design. 94--97.

Digital Library

[43]

K T Mullen. 1985. The contrast sensitivity of human colour vision to red-green and blue-yellow chromatic gratings. The Journal of Physiology 359, 1 (1985), 381--400.

[44]

Prathyusha Narra and D. S. Zinger. 2004. An effective LED dimming approach. In Conference Record of the 2004 IEEE Industry Applications Conference, 2004. 39th IAS Annual Meeting., Vol. 3.

[45]

I. Omer and M. Werman. 2004. Color lines: image specific color representation. IEEE CVPR (2004), 946--953.

[46]

Mang Ou-Yang and Shih-Wei Huang. 2007. Design considerations between color gamut and brightness for multi-primary color displays. Journal of Display Technology 3, 1 (2007), 71--82.

[47]

Hao Pan, Xiao-Fan Feng, and S. Daly. 2005. LCD motion blur modeling and analysis. In IEEE ICIP.

[48]

A. A. Polumordvinov. 1899. Russian Patent 10738. (1899).

[49]

Tania Pouli, Douglas W Cunningham, and Erik Reinhard. 2010. Image statistics and their applications in computer graphics. In European Computer Graphics Conference and Exhibition.

[50]

Joseph P. Rice, Steven W. Brown, David W. Allen, Howard W. Yoon, Maritoni Litorja, and Jeeseong C. Hwang. 2012. Hyperspectral image projector applications. Proc. SPIE 8254, 82540R--82540R-8.

[51]

Joseph P. Rice, Steven W. Brown, Jorge E. Neira, and Robert R. Bousquet. 2007. A hyperspectral image projector for hyperspectral imagers. Proc. SPIE 6565, 65650C--65650C-12.

[52]

Temkar N. Ruckmongathan. 2014. Addressing Techniques of Liquid Crystal Displays. Wiley.

[53]

Alfred C. Schroeder. 1948. Color television tube. U.S. Patent 2446791A. (1948).

[54]

Helge Seetzen, Wolfgang Heidrich, Wolfgang Stuerzlinger, Greg Ward, Lorne White-head, Matthew Trentacoste, Abhijeet Ghosh, and Andrejs Vorozcovs. 2004. High Dynamic Range Display Systems. ACM Trans. Graph. (SIGGRAPH) 23, 3 (2004), 760--768.

Digital Library

[55]

G. Sharma, W. Wu, and E. N. Dalal. 2005. The CIEDE2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations. Color research and application 30, 1 (2005), 21--30.

[56]

Louis D. Silverstein. 2005. STColor: Hybrid Spatial-Temporal Color Synthesis for Enhanced Display Image Quality. In SID Symposium Digest of Technical Papers, Vol. 36. 1112--1115.

[57]

Ernst Simonson and Josef Brozek. 1952. Flicker fusion frequency. Physiological Review 32, 3 (1952), 349--378.

[58]

Wen-Chih Tai, Chi-Chung Tsai, Shian-Jun Chiou, Chih-Ping Su, Huang-Min Chen, Chia-Lin Liu, and Chi-Neng Mo. 2008. Field Sequential Color LCD-TV Using Multi-Area Control Algorithm. SID Symposium Digest of Technical Papers 39, 1 (2008), 1092--1095.

[59]

Masatsugu Teragawa, Akiko Yoshida, Kazuyoshi Yoshiyama, Shinji Nakagawa, Kazunari Tomizawa, and Yasuhiro Yoshida. 2012. Review Paper: Multi-primary-color displays: The latest technologies and their benefits. Journal of the Soc. for Info. Disp. 20, 1 (2012).

[60]

Takatoshi Tsujimura. 2012. OLED Display Fundamentals and Applications. Wiley.

[61]

Rui Wang, Bowen Yu, Julio Marco, Tianlei Hu, Diego Gutierrez, and Hujun Bao. 2016. Real-time Rendering on a Power Budget. ACM Trans. Graph. 35, 4, Article 111 (July 2016), 11 pages.

Digital Library

[62]

Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.

Digital Library

[63]

G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar. 2012. Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting. ACM Trans. Graph. (SIGGRAPH) 31, 4 (2012), 1--11.

Digital Library

[64]

Lin Zhang, D. Zhang, Xuanqin Mou, and D. Zhang. 2011. FSIM: A Feature Similarity Index for Image Quality Assessment. IEEE Transactions on Image Processing 20, 8 (2011), 2378--2386.

Digital Library

[65]

X. Zhang and B. A. Wandell. 1997. A spatial extension of CIELAB for digital color-image reproduction. Journal of the Society for Information Display 5, 1 (1997), 61--63.

Cited By

Chen LWeng LCheng HCheng AHuang C(2025)Temporal Fusion: Continuous-Time Light Field Video FactorizationIEEE Transactions on Image Processing10.1109/TIP.2025.353320334(955-968)Online publication date: 2025
https://doi.org/10.1109/TIP.2025.3533203
Chen KWan TMatsuda NNinan AChapiro ASun Q(2024)PEA-PODs: Perceptual Evaluation of Algorithms for Power Optimization in XR DisplaysACM Transactions on Graphics10.1145/365812643:4(1-17)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658126
Chen LWeng LCheng HCheng ALin KHuang C(2024)VLSI Design of Light-Field Factorization for Dual-Layer Factored DisplayIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2024.341426232:11(2093-2106)Online publication date: Nov-2024
https://doi.org/10.1109/TVLSI.2024.3414262
Show More Cited By

Index Terms

Mixed-primary factorization for dual-frame computational displays
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Displays and imagers

Recommendations

A color compensation algorithm to avoid color distortion in active dimming liquid crystal displays

In LCD applications, various low power techniques, such as low voltage driving LEDs, high efficiency DC-DC converter circuitries, active backlight control schemes and the likes, have been applied to a backlight, which occupies the huge portion of total ...
Data-driven image color theme enhancement

It is often important for designers and photographers to convey or enhance desired color themes in their work. A color theme is typically defined as a template of colors and an associated verbal description. This paper presents a data-driven method for ...
Data-driven image color theme enhancement
SIGGRAPH ASIA '10: ACM SIGGRAPH Asia 2010 papers

It is often important for designers and photographers to convey or enhance desired color themes in their work. A color theme is typically defined as a template of colors and an associated verbal description. This paper presents a data-driven method for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 36, Issue 4

August 2017

2155 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3072959

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2017

Published in TOG Volume 36, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
469
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen LWeng LCheng HCheng AHuang C(2025)Temporal Fusion: Continuous-Time Light Field Video FactorizationIEEE Transactions on Image Processing10.1109/TIP.2025.353320334(955-968)Online publication date: 2025
https://doi.org/10.1109/TIP.2025.3533203
Chen KWan TMatsuda NNinan AChapiro ASun Q(2024)PEA-PODs: Perceptual Evaluation of Algorithms for Power Optimization in XR DisplaysACM Transactions on Graphics10.1145/365812643:4(1-17)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658126
Chen LWeng LCheng HCheng ALin KHuang C(2024)VLSI Design of Light-Field Factorization for Dual-Layer Factored DisplayIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2024.341426232:11(2093-2106)Online publication date: Nov-2024
https://doi.org/10.1109/TVLSI.2024.3414262
Kavaklı KShi LUrey HMatusik WAkşit K(2024)59‐1: Enhancing Brightness with Multi‐Color HolographySID Symposium Digest of Technical Papers10.1002/sdtp.1765155:1(809-812)Online publication date: 30-Jul-2024
https://doi.org/10.1002/sdtp.17651
Wang ZZou GLi YMa CZhang ZZha GShen YYang BQin Z(2023)Two-field sequential color liquid crystal displays with deep learning-enabled real-time drivingOptics Letters10.1364/OL.50156748:21(5779)Online publication date: 27-Oct-2023
https://doi.org/10.1364/OL.501567
Kavaklı KShi LUrey HMatusik WAkşit K(2023)Multi-color Holograms Improve Brightness in Holographic DisplaysSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618135(1-11)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618135
Zou GWang ZLiu YLi JLiu XLiu JYang BQin Z(2022)Deep learning-enabled image content-adaptive field sequential color LCDs with mini-LED backlightOptics Express10.1364/OE.45975230:12(21044)Online publication date: 26-May-2022
https://doi.org/10.1364/OE.459752
Yoo DLee SJo YCho JChoi SLee B(2022)Volumetric Head-Mounted Display With Locally Adaptive Focal BlocksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.301146828:2(1415-1427)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TVCG.2020.3011468
Banerjee ACamps JZacur EAndrews CRudy YChoudhury RRodriguez BGrau V(2021)A completely automated pipeline for 3D reconstruction of human heart from 2D cine magnetic resonance slicesPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences10.1098/rsta.2020.0257379:2212Online publication date: 25-Oct-2021
https://doi.org/10.1098/rsta.2020.0257
Wetzstein GPatney ASun Q(2020)State of the Art in Perceptual VR DisplaysReal VR – Immersive Digital Reality10.1007/978-3-030-41816-8_9(221-243)Online publication date: 3-Mar-2020
https://doi.org/10.1007/978-3-030-41816-8_9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents