Gang Xu
Ming-Ming Cheng
ABSTRACT
High-dynamic-range (HDR) media resources that preserve high contrast and more details in shadow and highlight areas in television are becoming increasingly popular for modern display technology compared to the widely available standard-dynamic-range (SDR) media resources. However, due to the exorbitant price of HDR cameras, researchers have attempted to develop the SDR-to-HDR techniques to convert the abundant SDR media resources to the HDR versions for cost-saving. Recent SDR-to-HDR methods mostly apply the image-adaptive modulation scheme to dynamically modulate the local contrast. However, these methods often fail to properly capture the low-frequency cues, resulting in artifacts in the low-frequency regions and low visual quality. Motivated by the Discrete Cosine Transform (DCT), in this paper, we propose a Frequency-aware Modulation Network (FMNet) to enhance the contrast in a frequency-adaptive way for SDR-to-HDR translation. Specifically, we design a frequency-aware modulation block that can dynamically modulate the features according to its frequency-domain responses. This allows us to reduce the structural distortions and artifacts in the translated low-frequency regions and reconstruct high-quality HDR content in the translated results. Experimental results on the HDRTV1K dataset show that our FMNet outperforms previous methods and the perceptual quality of the generated HDR images can be largely improved. Our code is available at https://github.com/MCG-NKU/FMNet.
Supplemental Material
- Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. 1974. Discrete cosine transform. IEEE transactions on Computers 100, 1 (1974), 90--93.Google Scholar
- Aman R Chadha, Pallavi P Vaidya, and M Mani Roja. 2011. Face recognition using discrete cosine transform for global and local features. In 2011 International Conference On Recent Advancements In Electrical, Electronics And Control Engineering. IEEE, 502--505.Google ScholarCross Ref
- Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. 2021. Amplitude-phase recombination: Rethinking robustness of convolutional neural networks in frequency domain. In Int. Conf. Comput. Vis. 458--467.Google ScholarCross Ref
- Wenlin Chen, James Wilson, Stephen Tyree, Kilian QWeinberger, and Yixin Chen. 2016. Compressing convolutional neural networks in the frequency domain. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1475--1484.Google ScholarDigital Library
- Xiangyu Chen, Zhengwen Zhang, Jimmy S. Ren, Lynhoo Tian, Yu Qiao, and Chao Dong. 2021. A New Journey from SDRTV to HDRTV. In Int. Conf. Comput. Vis.Google ScholarCross Ref
- Paul E. Debevec and Jitendra Malik. 2008. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH Anal. Conf.Google Scholar
- Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2015), 295--307.Google ScholarDigital Library
- Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. 36, 4 (2017), 1--12.Google ScholarDigital Library
- Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi. 2017. From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur. In IEEE Conf. Comput. Vis. Pattern Recog. 2319--2328.Google ScholarCross Ref
- Lionel Gueguen, Alex Sergeev, Ben Kadlec, Rosanne Liu, and Jason Yosinski. 2018. Faster neural networks straight from jpeg. Adv. Neural Inform. Process. Syst. 31 (2018).Google Scholar
- Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. 2020. Conditional sequential modulation for efficient global image retouching. In Eur. Conf. Comput. Vis. Springer, 679--695.Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conf. Comput. Vis. Pattern Recog. 770--778.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In Eur. Conf. Comput. Vis. Springer, 630--645.Google ScholarCross Ref
- Yongqing Huo, Fan Yang, Le Dong, and Vincent Brost. 2014. Physiological inverse tone mapping based on retina response. The Visual Computer 30, 5 (2014), 507--517.Google ScholarCross Ref
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Int. Conf. Mach. Learn. PMLR, 448--456.Google Scholar
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-toimage translation with conditional adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog. 1125--1134.Google Scholar
- Soo Ye Kim and Munchurl Kim. 2018. A multi-purpose convolutional neural network for simultaneous super-resolution and high dynamic range image reconstruction. In Asian Conf. Comput. Vis. Springer, 379--394.Google Scholar
- Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2019. Deep sr-itm: Joint learning of super-resolution and inverse tone-mapping for 4k uhd hdr applications. In Int. Conf. Comput. Vis. 3116--3125.Google ScholarCross Ref
- Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2020. Jsi-gan: Gan-based joint super-resolution and inverse tone-mapping with pixel-wise task-specific filters for uhd hdr video. In Association for the Advancement of Artificial Intelligence. 11287--11295.Google Scholar
- Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Int. Conf. Learn. Represent.Google Scholar
- Rafael P Kovaleski and Manuel M Oliveira. 2014. High-quality reverse tone mapping for a wide range of exposures. In 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images. IEEE, 49--56.Google ScholarDigital Library
- Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh. 136--144.Google ScholarCross Ref
- Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-image HDR reconstruction by learning to reverse the camera pipeline. In IEEE Conf. Comput. Vis. Pattern Recog. 1651--1660.Google ScholarCross Ref
- Zhenhua Liu, Jizheng Xu, Xiulian Peng, and Ruiqin Xiong. 2018. Frequencydomain dynamic pruning for convolutional neural networks. Adv. Neural Inform. Process. Syst. 31 (2018).Google Scholar
- Salma Abdel Magid, Yulun Zhang, Donglai Wei, Won-Dong Jang, Zudi Lin, Yun Fu, and Hanspeter Pfister. 2021. Dynamic high-pass filtering and multi-spectral attention for image super-resolution. In Int. Conf. Comput. Vis. 4288--4297.Google ScholarCross Ref
- Rafa? Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. 30, 4 (2011), 1--14.Google ScholarDigital Library
- Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In IEEE Conf. Comput. Vis. Pattern Recog. 3883--3891.Google ScholarCross Ref
- Zequn Qin, Pengyi Zhang, Fei Wu, and Xi Li. 2021. Fcanet: Frequency channel attention networks. In Int. Conf. Comput. Vis. 783--792.Google ScholarCross Ref
- Daniele Ravì, Miroslaw Bober, Giovanni Maria Farinella, Mirko Guarnera, and Sebastiano Battiato. 2016. Semantic segmentation of images exploiting DCT based features and random forest. Pattern Recogn. 52 (2016), 260--273.Google ScholarDigital Library
- Xing Shen, Jirui Yang, Chunbo Wei, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, Xiaoliang Cheng, and Kewei Liang. 2021. Dct-mask: Discrete cosine transform mask representation for instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog. 8720--8729.Google ScholarCross Ref
- Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and ZehanWang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE Conf. Comput. Vis. Pattern Recog. 1874--1883.Google ScholarCross Ref
- Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz. 2019. Pixel-adaptive convolutional neural networks. In IEEE Conf. Comput. Vis. Pattern Recog. 11166--11175.Google ScholarCross Ref
- Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. 2015. Learning a convolutional neural network for non-uniform motion blur removal. In IEEE Conf. Comput. Vis. Pattern Recog. 769--777.Google ScholarCross Ref
- IT Union. 2015. Recommendation ITU-R BT. 709--6. Electronic Publication (2015).Google Scholar
- IT Union. 2016. Recommendation ITU-R BT. 2100--2. Electronic Publication (2016).Google Scholar
- IT Union. 2019. Recommendation ITU-R BT. 2124-0. Electronic Publication (2019).Google Scholar
- Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. 2016. Cnnpack: Packing convolutional neural networks in the frequency domain. Adv. Neural Inform. Process. Syst. 29 (2016).Google Scholar
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600--612.Google ScholarDigital Library
- Jie Wei. 2002. Image segmentation based on situational DCT descriptors. Pattern Recogn. 23, 1--3 (2002), 295--302.Google Scholar
- Wenbin Xie, Dehua Song, Chang Xu, Chunjing Xu, Hui Zhang, and Yunhe Wang. 2021. Learning Frequency-aware Dynamic Network for Efficient Super- Resolution. In Int. Conf. Comput. Vis. 4308--4317.Google Scholar
- Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, and Lei Zhang. 2020. Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time. IEEE Trans. Pattern Anal. Mach. Intell. (2020).Google ScholarCross Ref
- Kai Zhang, Yawei Li,Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. 2021. Plug-and-play image restoration with deep denoiser prior. IEEE Trans. Pattern Anal. Mach. Intell. (2021).Google Scholar
- Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Trans. Image Process. 26, 7 (2017), 3142--3155.Google ScholarDigital Library
- Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In IEEE Conf. Comput. Vis. Pattern Recog. 3929--3938.Google ScholarCross Ref
- Lin Zhang and Hongyu Li. 2012. SR-SIM: A fast and high performance IQA index based on spectral residual. In IEEE Int. Conf. Image Process. IEEE, 1473--1476.Google ScholarCross Ref
- Yulun Zhang, Kunpeng Li, Kai Li, LichenWang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Eur. Conf. Comput. Vis. 286--301.Google ScholarCross Ref
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Int. Conf. Comput. Vis. 2223--2232.Google ScholarCross Ref
- Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. 2020. Delving Deeper into Anti-aliasing in ConvNets. In Brit. Mach. Vis. Conf.Google Scholar
Index Terms
- FMNet: Frequency-Aware Modulation Network for SDR-to-HDR Translation
Recommendations
Frequency-domain equalization for continuous phase modulation
Single-carrier frequency-domain equalization (SC-FDE) is proposed for continuous phase modulation (CPM). Two different discrete representations of the CPM signal are obtained. One is an orthogonal representation obtained from the Gram-Schmidt ...
Generalization of Orthogonal Frequency Division Multiplexing With Index Modulation
Recently, orthogonal frequency division multiplexing (OFDM) with index modulation (OFDM-IM) was proposed. By selecting a fixed number of subcarriers as active subcarriers to carry constellation symbols, the indices of these active subcarriers may carry ...
Mainlobe Width Reduction Using Linear and Nonlinear Frequency Modulation
ARTCOM '09: Proceedings of the 2009 International Conference on Advances in Recent Technologies in Communication and ComputingHigh Range Resolution can be achieved by various methods in a radar system. Simple pulse systems are narrow band, easiest and least expensive to maintain, but one can achieve only limited resolution. To achieve good range resolution pulse compression ...
Comments